Unable to connect kernel

[mmuru]

Description

Jupyter Enterprise Gateway 3.2.2
Python 3.9.16 using virtual environment
Using single server deployment to validate JEG with jupyter Lab (running another venv terminal) but no success with it.

Please, let me know how to fix this issue. I appreciate your help in advance.

Jupyter server and lab details

IPython          : 8.15.0
ipykernel        : 6.25.2
jupyter_client   : 8.3.1
jupyter_core     : 5.3.1
jupyter_server   : 2.7.3
jupyterlab       : 4.0.6
nbclient         : 0.8.0
nbconvert        : 7.8.0
nbformat         : 5.9.2
traitlets        : 5.10.0

[D 2023-09-22 22:47:13.134 EnterpriseGatewayApp] Searching ['/Users/muru/projects/joyent/jupyter/gateway/venv-gw/etc/jupyter', '/Users/muru/.jupyter', '/usr/local/etc/jupyter', '/etc/jupyter'] for config files
[D 2023-09-22 22:47:13.134 EnterpriseGatewayApp] Looking for jupyter_config in /etc/jupyter
[D 2023-09-22 22:47:13.134 EnterpriseGatewayApp] Looking for jupyter_config in /usr/local/etc/jupyter
[D 2023-09-22 22:47:13.134 EnterpriseGatewayApp] Looking for jupyter_config in /Users/muru/.jupyter
[D 2023-09-22 22:47:13.135 EnterpriseGatewayApp] Looking for jupyter_config in /Users/muru/projects/joyent/jupyter/gateway/venv-gw/etc/jupyter
[D 2023-09-22 22:47:13.135 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /etc/jupyter
[D 2023-09-22 22:47:13.135 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /usr/local/etc/jupyter
[D 2023-09-22 22:47:13.135 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /Users/muru/.jupyter
[D 2023-09-22 22:47:13.135 EnterpriseGatewayApp] Looking for jupyter_enterprise_gateway_config in /Users/muru/projects/joyent/jupyter/gateway/venv-gw/etc/jupyter
[D 230922 22:47:13 selector_events:54] Using selector: KqueueSelector
[I 2023-09-22 22:47:13.142 EnterpriseGatewayApp] Jupyter Enterprise Gateway 3.2.2 is available at http://127.0.0.1:8888
[D 2023-09-22 22:48:18.630 EnterpriseGatewayApp] Found kernel python3 in /Users/muru/projects/joyent/jupyter/gateway/venv-gw/share/jupyter/kernels
[D 2023-09-22 22:48:18.630 EnterpriseGatewayApp] Found kernel venv-gw in /Users/muru/Library/Jupyter/kernels
[I 230922 22:48:18 web:2344] 200 GET /api/kernelspecs (127.0.0.1) 1308.87ms
[I 230922 22:48:18 web:2344] 200 GET /api/kernels (127.0.0.1) 0.45ms
[D 2023-09-22 22:48:18.778 EnterpriseGatewayApp] Found kernel python3 in /Users/muru/projects/joyent/jupyter/gateway/venv-gw/share/jupyter/kernels
[D 2023-09-22 22:48:18.778 EnterpriseGatewayApp] Found kernel venv-gw in /Users/muru/Library/Jupyter/kernels
[I 230922 22:48:18 web:2344] 200 GET /api/kernelspecs (127.0.0.1) 2.96ms
[D 2023-09-22 22:48:23.771 EnterpriseGatewayApp] RemoteMappingKernelManager.start_kernel: python3, kernel_username: guest
[D 2023-09-22 22:48:23.779 EnterpriseGatewayApp] Instantiating kernel 'Python 3 Local' with process proxy: enterprise_gateway.services.processproxies.processproxy.LocalProcessProxy
[D 2023-09-22 22:48:23.786 EnterpriseGatewayApp] Starting kernel (async): ['/Users/muru/projects/joyent/jupyter/gateway/venv-gw/bin/python', '-m', 'ipykernel_launcher', '-f', '/Users/muru/Library/Jupyter/runtime/kernel-f9fb4804-93e2-4a96-af80-52358186cac2.json']
[D 2023-09-22 22:48:23.787 EnterpriseGatewayApp] Launching kernel: 'Python 3 Local' with command: ['/Users/muru/projects/joyent/jupyter/gateway/venv-gw/bin/python', '-m', 'ipykernel_launcher', '-f', '/Users/muru/Library/Jupyter/runtime/kernel-f9fb4804-93e2-4a96-af80-52358186cac2.json']
[D 2023-09-22 22:48:23.787 EnterpriseGatewayApp] BaseProcessProxy.launch_process() env: {'KERNEL_LAUNCH_TIMEOUT': '********', 'KERNEL_USERNAME': '********', 'KERNEL_WORKING_DIR': '********', 'KERNEL_GATEWAY': '********', 'KERNEL_ID': '********', 'KERNEL_LANGUAGE': '********', 'EG_IMPERSONATION_ENABLED': '********'}
[I 2023-09-22 22:48:23.796 EnterpriseGatewayApp] Local kernel launched on '192.168.1.73', pid: 30019, pgid: 30019, KernelID: f9fb4804-93e2-4a96-af80-52358186cac2, cmd: '['/Users/muru/projects/joyent/jupyter/gateway/venv-gw/bin/python', '-m', 'ipykernel_launcher', '-f', '/Users/muru/Library/Jupyter/runtime/kernel-f9fb4804-93e2-4a96-af80-52358186cac2.json']'
[D 2023-09-22 22:48:23.797 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57456
[D 2023-09-22 22:48:23.798 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57453
[I 2023-09-22 22:48:23.800 EnterpriseGatewayApp] Kernel started: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:23.800 EnterpriseGatewayApp] Kernel args: {'env': {'KERNEL_LAUNCH_TIMEOUT': '40', 'KERNEL_USERNAME': 'guest', 'KERNEL_WORKING_DIR': '/Users/muru/projects/joyent/jupyter/client'}, 'kernel_headers': {}, 'kernel_name': 'python3'}
[I 230922 22:48:23 web:2344] 201 POST /api/kernels (127.0.0.1) 31.95ms
[D 2023-09-22 22:48:23.815 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels
[W 2023-09-22 22:48:23.816 EnterpriseGatewayApp] No session ID specified
[D 2023-09-22 22:48:23.816 EnterpriseGatewayApp] Requesting kernel info from f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:23.816 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57452
[I 230922 22:48:23 web:2344] 200 GET /api/kernels (127.0.0.1) 0.52ms
[D 2023-09-22 22:48:23.824 EnterpriseGatewayApp] Found kernel python3 in /Users/muru/projects/joyent/jupyter/gateway/venv-gw/share/jupyter/kernels
[D 2023-09-22 22:48:23.825 EnterpriseGatewayApp] Found kernel venv-gw in /Users/muru/Library/Jupyter/kernels
[I 230922 22:48:23 web:2344] 200 GET /api/kernelspecs (127.0.0.1) 4.24ms
[D 2023-09-22 22:48:23.830 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels
[W 2023-09-22 22:48:23.831 EnterpriseGatewayApp] No session ID specified
[D 2023-09-22 22:48:23.831 EnterpriseGatewayApp] Waiting for pending kernel_info request
[D 2023-09-22 22:48:23.847 EnterpriseGatewayApp] Initializing websocket connection /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels
[W 2023-09-22 22:48:23.847 EnterpriseGatewayApp] No session ID specified
[D 2023-09-22 22:48:23.848 EnterpriseGatewayApp] Waiting for pending kernel_info request
[D 2023-09-22 22:48:26.753 EnterpriseGatewayApp] activity on f9fb4804-93e2-4a96-af80-52358186cac2: status (busy)
[D 2023-09-22 22:48:26.754 EnterpriseGatewayApp] activity on f9fb4804-93e2-4a96-af80-52358186cac2: status (idle)
[D 2023-09-22 22:48:26.755 EnterpriseGatewayApp] Received kernel info: {'status': 'ok', 'protocol_version': '5.3', 'implementation': 'ipython', 'implementation_version': '8.15.0', 'language_info': {'name': 'python', 'version': '3.9.16', 'mimetype': 'text/x-python', 'codemirror_mode': {'name': 'ipython', 'version': 3}, 'pygments_lexer': 'ipython3', 'nbconvert_exporter': 'python', 'file_extension': '.py'}, 'banner': "Python 3.9.16 (main, Mar  6 2023, 11:20:31) \nType 'copyright', 'credits' or 'license' for more information\nIPython 8.15.0 -- An enhanced Interactive Python. Type '?' for help.\n", 'help_links': [{'text': 'Python Reference', 'url': 'https://docs.python.org/3.9'}, {'text': 'IPython Reference', 'url': 'https://ipython.org/documentation.html'}, {'text': 'NumPy Reference', 'url': 'https://docs.scipy.org/doc/numpy/reference/'}, {'text': 'SciPy Reference', 'url': 'https://docs.scipy.org/doc/scipy/reference/'}, {'text': 'Matplotlib Reference', 'url': 'https://matplotlib.org/contents.html'}, {'text': 'SymPy Reference', 'url': 'http://docs.sympy.org/latest/index.html'}, {'text': 'pandas Reference', 'url': 'https://pandas.pydata.org/pandas-docs/stable/'}]}
[I 230922 22:48:26 web:2344] 101 GET /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels (127.0.0.1) 2941.40ms
[D 2023-09-22 22:48:26.757 EnterpriseGatewayApp] Opening websocket /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels
[D 2023-09-22 22:48:26.757 EnterpriseGatewayApp] Getting buffer for f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.757 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57453
[D 2023-09-22 22:48:26.757 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57452
[D 2023-09-22 22:48:26.758 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57456
[D 2023-09-22 22:48:26.758 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57454
[D 2023-09-22 22:48:26.758 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57452
[D 2023-09-22 22:48:26.759 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57456
[I 230922 22:48:26 web:2344] 101 GET /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels (127.0.0.1) 2929.28ms
[D 2023-09-22 22:48:26.760 EnterpriseGatewayApp] Opening websocket /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels
[D 2023-09-22 22:48:26.760 EnterpriseGatewayApp] Getting buffer for f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.760 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57453
[D 2023-09-22 22:48:26.760 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57452
[D 2023-09-22 22:48:26.760 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57456
[D 2023-09-22 22:48:26.761 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57454
[D 2023-09-22 22:48:26.761 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57452
[D 2023-09-22 22:48:26.761 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57456
[I 230922 22:48:26 web:2344] 101 GET /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels (127.0.0.1) 2914.92ms
[D 2023-09-22 22:48:26.762 EnterpriseGatewayApp] Opening websocket /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels
[D 2023-09-22 22:48:26.762 EnterpriseGatewayApp] Getting buffer for f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.762 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57453
[D 2023-09-22 22:48:26.762 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57452
[D 2023-09-22 22:48:26.763 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57456
[D 2023-09-22 22:48:26.763 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57454
[D 2023-09-22 22:48:26.763 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57452
[D 2023-09-22 22:48:26.763 EnterpriseGatewayApp] Connecting to: tcp://127.0.0.1:57456
[D 2023-09-22 22:48:26.765 EnterpriseGatewayApp] Nudge: attempt 1 on kernel f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.766 EnterpriseGatewayApp] Nudge: attempt 1 on kernel f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.766 EnterpriseGatewayApp] Nudge: attempt 1 on kernel f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.769 EnterpriseGatewayApp] activity on f9fb4804-93e2-4a96-af80-52358186cac2: status (busy)
[D 2023-09-22 22:48:26.769 EnterpriseGatewayApp] Nudge: IOPub received: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.769 EnterpriseGatewayApp] Nudge: resolving iopub future: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.769 EnterpriseGatewayApp] Nudge: IOPub received: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.769 EnterpriseGatewayApp] Nudge: resolving iopub future: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.770 EnterpriseGatewayApp] Nudge: IOPub received: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.770 EnterpriseGatewayApp] Nudge: resolving iopub future: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.770 EnterpriseGatewayApp] activity on f9fb4804-93e2-4a96-af80-52358186cac2: status (idle)
[D 2023-09-22 22:48:26.771 EnterpriseGatewayApp] activity on f9fb4804-93e2-4a96-af80-52358186cac2: status (busy)
[D 2023-09-22 22:48:26.771 EnterpriseGatewayApp] Nudge: shell info reply received: f9fb4804-93e2-4a96-af80-52358186cac2
[D 2023-09-22 22:48:26.771 EnterpriseGatewayApp] Nudge: resolving shell future: f9fb4804-93e2-4a96-af80-52358186cac2
ws_msg: 8=shell{"date":"2023-09-23T05:48:23.814Z","msg_id":"6db80b74-eeb0-40a0-96f0-dc875651ef86","msg_type":"kernel_info_request","session":"e953b98f-03ba-4541-a218-e12a880c7940","username":"","version":"5.2"}{}{}{}
[E 230922 22:48:26 web:1871] Uncaught exception GET /api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels (127.0.0.1)
    HTTPServerRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/api/kernels/f9fb4804-93e2-4a96-af80-52358186cac2/channels', version='HTTP/1.1', remote_ip='127.0.0.1')
    Traceback (most recent call last):
      File "/Users/muru/projects/joyent/jupyter/gateway/venv-gw/lib/python3.9/site-packages/tornado/websocket.py", line 630, in _run_callback
        result = callback(*args, **kwargs)
      File "/Users/muru/projects/joyent/jupyter/gateway/venv-gw/lib/python3.9/site-packages/jupyter_server/services/kernels/handlers.py", line 522, in on_message
        msg = json.loads(ws_msg)
      File "/Users/muru/.pyenv/versions/3.9.16/lib/python3.9/json/__init__.py", line 346, in loads
        return _default_decoder.decode(s)
      File "/Users/muru/.pyenv/versions/3.9.16/lib/python3.9/json/decoder.py", line 337, in decode
        obj, end = self.raw_decode(s, idx=_w(s, 0).end())
      File "/Users/muru/.pyenv/versions/3.9.16/lib/python3.9/json/decoder.py", line 355, in raw_decode
        raise JSONDecodeError("Expecting value", s, err.value) from None
    json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
[D 2023-09-22 22:48:26.781 EnterpriseGatewayApp] Websocket closed f9fb4804-93e2-4a96-af80-52358186cac2:3bc1606c-b8ce165ac41533da8d57fb0c
[W 230922 22:48:26 zmqstream:568] Got events for closed stream <zmq.eventloop.zmqstream.ZMQStream object at 0x11acdce50>

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[lresende]

Could be this?

enterprise_gateway/pyproject.toml

Line 29 in 9424e3f

"jupyter_client>=6.1.12,<7", # Remove cap once EG supports kernel provisioners

[kevin-bates]

Correct. EG doesn't support jupyter_client 7 or higher, nor does it support jupyter_server 2 or higher.

If you must use server >= 2 and client >= 7 you may want to explore using gateway_provisioners.

If you need your "kernel server" separated from your notebook server, you could deploy kernel gateway with gateway provisioners to get similar functionality. That said, EG should be usable so long as your python environments are properly configured.

[mmuru]

@kevin-bates and @lresende - Thank you for your information. Below is my setup

1. ven-gw - virtual environment for JEG (3.2.2) and configured with a ipython kernel using pip

ipykernel==6.25.2
ipython==8.15.0
jupyter-client==6.1.12
jupyter-server==1.24.0
jupyter_core==5.3.1
jupyter_enterprise_gateway==3.2.2
jupyterlab-pygments==0.2.2
tornado==6.3.3
pyzmq==24.0.1

2. venv-nb - a virtual environment of jupyter-server & lab

Are you saying that I should use the same versions of jupyter-server and jupyter_client as JEG pinned to? i.e 6.1.12 and 1.24.0. Please, clarify it? If possibel, please share requirements.txt file?

Yes, I tried to use JEG on k8s (AWS-EKS) and Jupyter-server/lab from the laptop/EC2 box but I ran into the same above issues.

[I 2023-09-23 11:49:18.217 ServerApp] Connecting to kernel 448500a4-f6a7-437e-9cac-ea134712f108.
[I 2023-09-23 11:49:18.217 ServerApp] Connecting to ws://127.0.0.1:8888/api/kernels/448500a4-f6a7-437e-9cac-ea134712f108/channels
[W 2023-09-23 11:49:19.051 ServerApp] Lost connection to Gateway: 448500a4-f6a7-437e-9cac-ea134712f108
[I 2023-09-23 11:49:19.051 ServerApp] Attempting to re-establish the connection to Gateway in 1.49 secs (1/5): 448500a4-f6a7-437e-9cac-ea134712f108

I will test your suggestions and keep posted. Thanks again for your help.

[mmuru]

I was able to make it work using JupyterLab==3.6.5 with jupyter-client==6.1.12 and jupyter-server==1.24.0 same as JEG 3.2.2. Below is the details configuration.

jupyter-client==6.1.12
jupyter-events==0.7.0
jupyter-lsp==2.2.0
jupyter-server==1.24.0
jupyter-telemetry==0.1.0
jupyter-ydoc==0.2.5
jupyter_core==5.3.1
jupyter_server_fileid==0.9.0
jupyter_server_ydoc==0.8.0
jupyterhub==4.0.2
jupyterlab==3.6.5
jupyterlab-pygments==0.2.2
jupyterlab_server==2.25.0

Next, I will try suggested using kernel gateway with gateway provisioners. Is there an ETA for JEG supports kernel provisioners?

[kevin-bates]

Is there an ETA for JEG supports kernel provisioners?

Not really (unfortunately), unless someone wants to take that on. Although not required, it would be good to first move the base Jupyter Server to 2.0 prior to removing the process-proxies, but given the authorization hurdles encountered in a similar exercise in JKG, it might be better treat the two tasks independently. (Although if the goal is to have a combined Python env for both server and EG, we'd need JS 2.x. I don't view the combined envs to be a requirement since the whole purpose of EG is to separate the compute (kernels) from the user's desktop - and you can achieve that today using JKG and GatewayProvisioners.)