[EG4] Rely on ServerApp and Remote Kernel Provisioners

[echarles]

This issue tracks the design to:

1. Run EG on the jupyter-server ServerApp
2. Use the Remote Kernel Provisioners (see roadmap, remote provisioners and jupyter-client documentation.

cc/ @kevin-bates

[echarles]

@kevin-bates While upgrading my remote provisionner on Kubernetes setup from 2.5.0 to 3.1.0, I struggle with the public key. I have updated the pod spec (adding the PUBLIC_KEY env) and the image (adding `/home/jovyan/.ssh/id_rsa.pub) but the kernel pod fails with

/usr/local/bin/bootstrap-kernel.sh: line 20: kernel_class_option: command not found
+ python /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py --kernel-id a8a9a206-4dd3-4d51-b0a2-d8a1865bfbf2 --port-range 0..0 --response-address 172.17.0.18:8877 --public-key /home/jovyan/.ssh/id_rsa.pub --spark-context-initialization-mode none
[D 2022-12-22 16:40:16,550.550 launch_ipykernel] Using connection file '/tmp/kernel-a8a9a206-4dd3-4d51-b0a2-d8a1865bfbf2_ar65gfac.json'.
[I 2022-12-22 16:40:16,552.552 launch_ipykernel] Signal socket bound to host: 0.0.0.0, port: 39247
[D 2022-12-22 16:40:16,553.553 launch_ipykernel] JSON Payload 'b'{"shell_port": 54727, "iopub_port": 56257, "stdin_port": 57735, "control_port": 49063, "hb_port": 52077, "ip": "0.0.0.0", "key": "094e7feb-498a-4211-b708-94e8936a11a9", "transport": "tcp", "signature_scheme": "hmac-sha256", "kernel_name": "", "pid": 10, "pgid": 7, "comm_port": 39247, "kernel_id": "a8a9a206-4dd3-4d51-b0a2-d8a1865bfbf2"}'
Traceback (most recent call last):
  File "/usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py", line 615, in <module>
    comm_socket = return_connection_info(
  File "/usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py", line 272, in return_connection_info
    payload = _encrypt(json_content, public_key)
  File "/usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py", line 217, in _encrypt
    imported_public_key = RSA.importKey(base64.b64decode(public_key.encode()))
  File "/opt/conda/lib/python3.9/base64.py", line 87, in b64decode
    return binascii.a2b_base64(s)

I guess kernel_class_option: command not found is relevant, may the cause of the subsequent errors?

[echarles]

PS: Before tackling this issue, I am upgrading my experiments to have a working basis.

[kevin-bates]

Before tackling this issue, I am upgrading my experiments to have a working basis.

Sounds good. I wanted to add a couple of points for clarification though.

1. While upgrading my remote provisioner

   Please note that EG has nothing to do with remote or kernel provisioners in general. I notice the [EG4] reference in the title and, had this been an issue relative to remote provisioners, it would be redirected to the appropriate repository since EG 4.0 (will be, and as Kernel Gateway is now) are merely clients of kernel provisioners.

2. The public key that is likely the target of the issue has nothing to do with /home/jovyan/.ssh/id_rsa.pub. Since support for the single response address was added, EG creates the Public/Private key pairs at startup, holding onto them only for the duration of when it's running, then will create another pair upon restart. So, they are ephemeral and never persisted.

3. I guess kernel_class_option: command not found is relevant, may the cause of the subsequent errors?

   This is certainly a sign that your launchers (likely in your kernel images if they are custom) are out of date.

In the remote provisioners repo, I believe we make it easier to build your images since it provides tooling for installing things like the bootstrap framework into images. Stay tuned.

I'm essentially "offline" until January 3 but will check in on occasion, especially if the issue looks blocking.

[echarles]

Thx for the help @kevin-bates. I have updated the kernelspec to add the public-key param and now it works. Have a happy end of year.

[kevin-bates]

Glad to hear you're moving forward. I just realized this was an existing issue and not newly created. Since this particular issue is more of an "umbrella issue", we should probably create a new issue if something else comes up. Take care and happy holidays!