Summary
jupyter-server-proxy
is used to expose ports local to a Jupyter server listening to web traffic to the Jupyter server's authenticated users by proxying web requests and websockets. Dependent packages (partial list) also use jupyter-server-proxy
to expose other popular interactive applications (such as RStudio, Linux Desktop via VNC, Code Server, Panel, etc) along with the Jupyter server. This feature is commonly used in hosted environments (such as a JupyterHub) to expose non-Jupyter interactive frontends or APIs to the user.
jupyter-server-proxy
did not check user authentication appropriately when proxying websockets, allowing unauthenticated access to anyone who had network access to the Jupyter server endpoint.
Impact
This vulnerability can allow unauthenticated remote access to any websocket endpoint set up to be accessible via jupyter-server-proxy
. In many cases (such as when exposing RStudio via jupyter-rsession-proxy
or a remote Linux Desktop / VNC via jupyter-remote-desktop-proxy
), this leads to remote unauthenticated arbitrary code execution, due to how they use websockets. The websocket endpoints exposed by jupyter_server
itself is not affected. Projects that do not rely on websockets are also not affected.
Remediation
Upgrade jupyter-server-proxy
to a patched version and restart any running Jupyter server.
You may not be installing jupyter-server-proxy
directly, but have it be pulled in as a dependency (partial list of dependent packages) - so you may be vulnerable even if you aren't directly depending on jupyter-server-proxy
.
For JupyterHub admins of TLJH installations
Expand to read more
To secure a tljh deployment's user servers, first check if jupyter-server-proxy
is installed in the user environment with a vulnerable version. If it is, patch the vulnerability and consider terminating currently running user servers.
1. Check for vulnerability
As an JupyterHub admin from a terminal in a started user server, you can do:
sudo -E python3 -c '
try:
import jupyter_server_proxy
is_vulnerable = not hasattr(jupyter_server_proxy, "__version__")
except:
is_vulnerable = False
if is_vulnerable:
print("WARNING: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.")
else:
print("INFO: not vulnerable to GHSA-w3vc-fx9p-wp4v")
'
Alternatively as a root user on the server where tljh is installed, you can do:
sudo PATH=/opt/tljh/user/bin:${PATH} python3 -c '
try:
import jupyter_server_proxy
is_vulnerable = not hasattr(jupyter_server_proxy, "__version__")
except:
is_vulnerable = False
if is_vulnerable:
print("WARNING: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.")
else:
print("INFO: not vulnerable to GHSA-w3vc-fx9p-wp4v")
'
2. Patch detected vulnerability
As an JupyterHub admin from a terminal in a started user server, you can do:
sudo -E pip install "jupyter-server-proxy>=3.2.3,!=4.0.0,!=4.1.0"
Alternatively as a root user on the server where tljh is installed, you can do:
sudo PATH=/opt/tljh/user/bin:${PATH} pip install "jupyter-server-proxy>=3.2.3,!=4.0.0,!=4.1.0"
3. Consider terminating currently running user servers
User servers that started before the patch was applied are still vulnerable. To ensure they aren't vulnerable any more you could forcefully terminate their servers via the JupyterHub web interface at https://<your domain>/hub/admin
.
For JupyterHub admins of Z2JH installations
Expand to read more
To secure your z2jh deployment's user servers, first consider if one or more user environments is or may be vulnerable, then ensure new user servers' aren't started with the vulnerability, and finally consider terminating currently running user servers. The steps below guide you to do so.
1. Check for vulnerabilities
Consider all docker images that user servers' environment may be based on. If your deployment expose a fixed set of images, you may be able to update them to non-vulnerable versions.
To check if an individual docker image is vulnerable, use a command like:
CHECK_IMAGE=jupyter/base-notebook:2023-10-20
docker run --rm $CHECK_IMAGE python3 -c '
try:
import jupyter_server_proxy
is_vulnerable = not hasattr(jupyter_server_proxy, "__version__")
except:
is_vulnerable = False
if is_vulnerable:
print("WARNING: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.")
else:
print("INFO: not vulnerable to GHSA-w3vc-fx9p-wp4v")
'
Note that if you reference an image with a mutable tag, such as quay.io/jupyter/pangeo-notebook:master
, you should ensure a new version is used by configuring the image pull policy so that an older vulnerable version isn't kept being used because it was already available on a Kubernetes node.
singleuser:
image:
name: quay.io/jupyter/pangeo-notebook
tag: master
# pullPolicy (a.k.a. imagePullPolicy in k8s specification) should be
# declared to Always if you make use of mutable tags
pullPolicy: Always
2. Patch vulnerabilities dynamically
If your z2jh deployment still may start vulnerable images for users, you could mount a script that checks and patches the vulnerability before the jupyter server starts.
Below is JupyterHub Helm chart configuration that relies on singleuser.extraFiles
and singleuser.cmd
to mount a script we use as an entrypoint to dynamically check and patch the vulnerability before jupyter server is started.
Unless you change it, the script will attempt to upgrade jupyter-server-proxy
to a non-vulnerable version if needed, and error if it needs to and fails. You can adjust this behavior by adjusting the constants UPGRADE_IF_VULNERABLE
and ERROR_IF_VULNERABLE
inside the script.
singleuser:
cmd:
- /mnt/ghsa-w3vc-fx9p-wp4v/check-patch-run
- jupyterhub-singleuser
extraFiles:
ghsa-w3vc-fx9p-wp4v-check-patch-run:
mountPath: /mnt/ghsa-w3vc-fx9p-wp4v/check-patch-run
mode: 0755
stringData: |
#!/usr/bin/env python3
"""
This script is designed to check for and conditionally patch GHSA-w3vc-fx9p-wp4v
in user servers started by a JupyterHub. The script will execute any command
passed via arguments if provided, allowing it to wrap a user server startup call
to `jupyterhub-singleuser` for example.
Use and function of this script can be further discussed in
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/3360.
Script adjustments:
- UPGRADE_IF_VULNERABLE
- ERROR_IF_VULNERABLE
Script patching assumptions:
- script is run before the jupyter server starts
- pip is available
- pip has sufficient filesystem permissions to upgrade jupyter-server-proxy
Read more at https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.
"""
import os
import subprocess
import sys
# adjust these to meet vulnerability mitigation needs
UPGRADE_IF_VULNERABLE = True
ERROR_IF_VULNERABLE = True
def check_vuln():
"""
Checks for the vulnerability by looking to see if __version__ is available
as it coincides with the patched versions (3.2.3 and 4.1.1).
"""
try:
import jupyter_server_proxy
return False if hasattr(jupyter_server_proxy, "__version__") else True
except:
return False
def get_version_specifier():
"""
Returns a pip version specifier for use with `--no-deps` meant to do as
little as possible besides patching the vulnerability and remaining
functional.
"""
old = ["jupyter-server-proxy>=3.2.3,<4"]
new = ["jupyter-server-proxy>=4.1.1,<5", "simpervisor>=1,<2"]
try:
if sys.version_info < (3, 8):
return old
from importlib.metadata import version
jsp_version = version("jupyter-server-proxy")
if int(jsp_version.split(".")[0]) < 4:
return old
except:
pass
return new
def patch_vuln():
"""
Attempts to patch the vulnerability by upgrading jupyter-server-proxy using
pip. Returns True if the patch is applied successfully, otherwise False.
"""
# attempt upgrade via pip, takes ~4 seconds
proc = subprocess.run(
[sys.executable, "-m", "pip", "--version"],
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
)
pip_available = proc.returncode == 0
if pip_available:
proc = subprocess.run(
[sys.executable, "-m", "pip", "install", "--no-deps"]
+ get_version_specifier()
)
if proc.returncode == 0:
return True
return False
def main():
if check_vuln():
warning_or_error = (
"ERROR" if ERROR_IF_VULNERABLE and not UPGRADE_IF_VULNERABLE else "WARNING"
)
print(
f"{warning_or_error}: jupyter-server-proxy __is vulnerable__ to GHSA-w3vc-fx9p-wp4v, see "
"https://github.com/jupyterhub/jupyter-server-proxy/security/advisories/GHSA-w3vc-fx9p-wp4v.",
flush=True,
)
if warning_or_error == "ERROR":
sys.exit(1)
if UPGRADE_IF_VULNERABLE:
print(
"INFO: Attempting to upgrade jupyter-server-proxy using pip...",
flush=True,
)
if patch_vuln():
print(
"INFO: Attempt to upgrade jupyter-server-proxy succeeded!",
flush=True,
)
else:
warning_or_error = "ERROR" if ERROR_IF_VULNERABLE else "WARNING"
print(
f"{warning_or_error}: Attempt to upgrade jupyter-server-proxy failed!",
flush=True,
)
if warning_or_error == "ERROR":
sys.exit(1)
if len(sys.argv) >= 2:
print("INFO: Executing provided command", flush=True)
os.execvp(sys.argv[1], sys.argv[1:])
else:
print("INFO: No command to execute provided", flush=True)
main()
3. Consider terminating currently running user servers
User servers that started before the patch was applied are still vulnerable. To ensure they aren't vulnerable any more you could forcefully terminate their servers via the JupyterHub web interface at https://<your domain>/hub/admin
.
Simple Reproduction
Expand to read more
Setup application to proxy
Make a trivial tornado app that has both websocket and regular HTTP endpoints.
from tornado import websocket, web, ioloop
class EchoWebSocket(websocket.WebSocketHandler):
def open(self):
print("WebSocket opened")
def on_message(self, message):
self.write_message(u"You said: " + message)
def on_close(self):
print("WebSocket closed")
class HiHandler(web.RequestHandler):
def get(self):
self.write("Hi")
app = web.Application([
(r'/ws', EchoWebSocket),
(r'/hi', HiHandler)
])
if __name__ == '__main__':
app.listen(9500)
ioloop.IOLoop.instance().start()
Setup a clean environment with jupyter-server-proxy
and start a jupyter server
instance
We don't need jupyterlab or anything else here, just jupyter-server-proxy
would do.
python -m venv clean-env/
source clean-env/bin/activate
pip install jupyter-server-proxy
jupyter server
Verify HTTP requests require authentication
curl -L http://127.0.0.1:8888/proxy/9500/hi
This does not return the Hi
response, as expected. Instead, you get the HTML response asking for a token.
This is secure as intended.
Verify websocket requests doesn't authentication
The example makes use of websocat to test websockets. You can use any other tool you are familiar with too.
websocat ws://localhost:8888/proxy/9500/ws
At the terminal, type 'Just testing' and press Enter. You'll get You said: Just testing
without any authentication required.
Summary
jupyter-server-proxy
is used to expose ports local to a Jupyter server listening to web traffic to the Jupyter server's authenticated users by proxying web requests and websockets. Dependent packages (partial list) also usejupyter-server-proxy
to expose other popular interactive applications (such as RStudio, Linux Desktop via VNC, Code Server, Panel, etc) along with the Jupyter server. This feature is commonly used in hosted environments (such as a JupyterHub) to expose non-Jupyter interactive frontends or APIs to the user.jupyter-server-proxy
did not check user authentication appropriately when proxying websockets, allowing unauthenticated access to anyone who had network access to the Jupyter server endpoint.Impact
This vulnerability can allow unauthenticated remote access to any websocket endpoint set up to be accessible via
jupyter-server-proxy
. In many cases (such as when exposing RStudio viajupyter-rsession-proxy
or a remote Linux Desktop / VNC viajupyter-remote-desktop-proxy
), this leads to remote unauthenticated arbitrary code execution, due to how they use websockets. The websocket endpoints exposed byjupyter_server
itself is not affected. Projects that do not rely on websockets are also not affected.Remediation
Upgrade
jupyter-server-proxy
to a patched version and restart any running Jupyter server.You may not be installing
jupyter-server-proxy
directly, but have it be pulled in as a dependency (partial list of dependent packages) - so you may be vulnerable even if you aren't directly depending onjupyter-server-proxy
.For JupyterHub admins of TLJH installations
Expand to read more
To secure a tljh deployment's user servers, first check if
jupyter-server-proxy
is installed in the user environment with a vulnerable version. If it is, patch the vulnerability and consider terminating currently running user servers.1. Check for vulnerability
As an JupyterHub admin from a terminal in a started user server, you can do:
Alternatively as a root user on the server where tljh is installed, you can do:
2. Patch detected vulnerability
As an JupyterHub admin from a terminal in a started user server, you can do:
sudo -E pip install "jupyter-server-proxy>=3.2.3,!=4.0.0,!=4.1.0"
Alternatively as a root user on the server where tljh is installed, you can do:
3. Consider terminating currently running user servers
User servers that started before the patch was applied are still vulnerable. To ensure they aren't vulnerable any more you could forcefully terminate their servers via the JupyterHub web interface at
https://<your domain>/hub/admin
.For JupyterHub admins of Z2JH installations
Expand to read more
To secure your z2jh deployment's user servers, first consider if one or more user environments is or may be vulnerable, then ensure new user servers' aren't started with the vulnerability, and finally consider terminating currently running user servers. The steps below guide you to do so.
1. Check for vulnerabilities
Consider all docker images that user servers' environment may be based on. If your deployment expose a fixed set of images, you may be able to update them to non-vulnerable versions.
To check if an individual docker image is vulnerable, use a command like:
Note that if you reference an image with a mutable tag, such as
quay.io/jupyter/pangeo-notebook:master
, you should ensure a new version is used by configuring the image pull policy so that an older vulnerable version isn't kept being used because it was already available on a Kubernetes node.2. Patch vulnerabilities dynamically
If your z2jh deployment still may start vulnerable images for users, you could mount a script that checks and patches the vulnerability before the jupyter server starts.
Below is JupyterHub Helm chart configuration that relies on
singleuser.extraFiles
andsingleuser.cmd
to mount a script we use as an entrypoint to dynamically check and patch the vulnerability before jupyter server is started.Unless you change it, the script will attempt to upgrade
jupyter-server-proxy
to a non-vulnerable version if needed, and error if it needs to and fails. You can adjust this behavior by adjusting the constantsUPGRADE_IF_VULNERABLE
andERROR_IF_VULNERABLE
inside the script.3. Consider terminating currently running user servers
User servers that started before the patch was applied are still vulnerable. To ensure they aren't vulnerable any more you could forcefully terminate their servers via the JupyterHub web interface at
https://<your domain>/hub/admin
.Simple Reproduction
Expand to read more
Setup application to proxy
Make a trivial tornado app that has both websocket and regular HTTP endpoints.
Setup a clean environment with
jupyter-server-proxy
and start ajupyter server
instanceWe don't need jupyterlab or anything else here, just
jupyter-server-proxy
would do.python -m venv clean-env/ source clean-env/bin/activate pip install jupyter-server-proxy jupyter server
Verify HTTP requests require authentication
This does not return the
Hi
response, as expected. Instead, you get the HTML response asking for a token.This is secure as intended.
Verify websocket requests doesn't authentication
The example makes use of websocat to test websockets. You can use any other tool you are familiar with too.
At the terminal, type 'Just testing' and press Enter. You'll get
You said: Just testing
without any authentication required.