Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Meeting Notes 2024 #57

Open
Zsailer opened this issue Jan 4, 2024 · 47 comments
Open

Meeting Notes 2024 #57

Zsailer opened this issue Jan 4, 2024 · 47 comments

Comments

@Zsailer
Copy link
Member

Zsailer commented Jan 4, 2024 •
edited
Loading

Hello everyone,

Gitter

Welcome to the Jupyter Server Team!

We meet on Thursdays at 8:00am, Pacific Standard Time on Jupyter's Zoom Channel.You can add yourself to the weekly agenda here. Everyone is welcome!

Let's avoid using this thread for discussion. If you'd like to discuss something in the minutes, open a separate issue and reference this thread.

You can find previous year's notes here: 2020, 2021, 2022, 2023

Meeting Notes
@Zsailer Zsailer pinned this issue Jan 4, 2024
@Zsailer Zsailer mentioned this issue Jan 4, 2024
Closed
@Zsailer
Copy link
Member Author

Zsailer commented Jan 4, 2024

January 4th, 2024

Name affiliation username
Zach Sailer Apple Zsailer
Johan Mabille QuantStack @JohanMabille
Andrii Ieroshenko AWS @andrii-i
Ian Thomas QuantStack @ianthomas23
David Brochart QuantStack @davidbrochart

Agenda

  • Zach
    • New year, new meeting thread!
    • Membership maintenance check closes tomorrow.
      • Thank you, everyone, for a great year!
    • Port kernel gateway to Jupyter Server 2.x
      • Thank you, Kevin, for doing a majority of the work! 👏
      • Requires a major release (3.x) and maintenance of the old branch (2.x)
      • I will announce on discourse once released.
      • Ran into issues getting the websocket subprotocol working, so the Gateway client defaults to the legacy protocol.
    • Would anyone else like to moderate the Jupyter Server meeting?
    • Also, new year means we should run a fresh election for Jupyter Server representative.

@Zsailer
Copy link
Member Author

Zsailer commented Jan 18, 2024

January 11th, 2024

Name affiliation GitHub username
Zach Sailer Apple Zsailer
Vidar Fauske JP Morgan Chase vidartf
Jason Grout Databricks @jasongrout
Sylvain Corlay QuantStack @QuantStack
Ian Thomas QuantStack @ianthomas23
David Brochart QuantStack @davidbrochart
Johan Mabille QuantStack @JohanMabille
Andrii Ieroshenko AWS @andrii-i

Agenda

  • Zach

    • schema.jupyter.org progress
      • Add first set of schemas and a simple Python package for distributing schemas jupyter/schema#1
      • Requirements for this repo
        • Installs schemas on disk somewhere (python package?)
          • The current PR uses a Python package
        • Makes schemas available from schema.jupyter.org (permanent URLs)
        • Import schemas at runtime (in any language, in principle)
        • Generate types from schemas in any language, in principle
        • CLI as a Jupyter core application
      • Question: what schemas should go here?
        • event schemas from subprojects
        • specs for connection_file
        • specs for the kernelspec
        • specs for the kernel protocol
        • specs for jupyter mimetypes
        • jupyter_server OpenAPI spec
          • some discussion around this point.
          • there is a difference between the API surface and what the handlers do in response to that API. Maybe we can start by formalizing the API service.
      • Ownership of schemas
        • Which subproject owns this repo?
          • [@jasongrout] Foundations and Standards seems like the natural home
        • Is this the source of truth for schemas? For example, is nbformat the home for the nbformat schema, and it's copied here, or does the official schema live here?
          • [@jasongrout] It's nice having nbformat schema co-located with the tools that work with it, for example. Though I suppose it's not too much of a stretch to have the nbformat package depend on the schemas package.
      • Package versioning
        • Do we ship a separate package for each schema? Or do we bump the package for any schema update?

  • Sylvain / Ian: Subshells

    • Add JEP for sub-shells jupyter/enhancement-proposals#91
    • Ian is planning on summarizing the current state of the work over the last few years
    • Two main approaches have been discussed
      • "Subshells": A single thread running a message router to different execution threads
        • Everyone's mental modeling of threading works
      • "Dependent kernels": Multiple execution threads, each with their own ZMQ channels (except control channel is shared), so it appears to the outside world as a collection of independent kernels
        • Pretend like it's a kernel and things "just work"
        • Except lifecycle is slightly more complicated because control thread is shared. For example, what does it mean to shutdown a kernel? Do we shut down the single thread, or the whole process?
    • Ian showed demos of the "dependent kernels" approach
    • There was discussion about how multiple threads interact with status messages. Right now status messages are sent for control message processing, which breaks the mental model for status messages.
    • Are there concerns about multiple threads interacting with the same display objects? Is it possible to do it in a thread safe manner? Or should diplay objects be owned by specific threads? What is the case for multiple clients?
      • [@jasongrout] I used to think the GIL protects us there, but some things I read recently helped me realize Python multithreading can be really tricky. (1, )
        • PEP 703 maybe not always rely on GIL?
    • Possibly in the future, status messages indicate busy/idle status for a single execution thread.
    • What's next?
      • QuantStack has funding to work on this, and is planning on working through the Jupyter process in the next 4-6 months with implementation landing within a year
      • We need to decide the basic approach, so have a discussion and move forward

  • David

    • YDrive: CRDT-based contents API
    • Will implement something for JupyterLab for this collaborative drive. Starting today.

@Zsailer
Copy link
Member Author

Zsailer commented Jan 25, 2024

January 18th, 2024

Name affiliation GitHub username
Zach Sailer Apple Zsailer
Johan Mabille QuantStack @JohanMabille
Jason Grout Databricks @jasongrout
Mike Krassowski Quansight @krassowski
Vidar Fauske JP Morgan Chase @vidartf
Ian Thomas QuantStack @ianthomas23
David Brochart QuantStack @davidbrochart
Andrii Ieroshenko AWS @andrii-i
William Stein SageMath, Inc. @williamstein

  • Mike

  • Ian Thomas

    • Update on concurrent kernels (subshells)
    • Add JEP for sub-shells jupyter/enhancement-proposals#91 (comment)
    • We spent ~5 minutes reading the newest update then opened for discussion.
    • Could it be that different dependent kernels have different ip addresses, i.e., totally different connection files?
      • Ian was thinking the connection file for the new dependent kernel just has a different shell port
      • If you have a different ip address, dependent kernels becomes multi-processing instead of multi-threaded
    • Dependent kernels require quite a bit of server changes
    • If we make the feature optional, the dependent kernels may be easier since you don't have to modify the kernel protocol
    • How backwards compatible is the approach for clients that support concurrent kernels, but kernels that don't?
      • subshells: the kernel just ignores the subshell field and messages are considered in order. The guarantee is that messages in a single subshell are processed in order.
      • dependent kernels: the api just hands back
    • the big question i have is how this kernel metadata stored in the notebook format? the notebook document stores compute information in its metadata. will subshells be marked up in the cell metadata to describe compute?
      • For many users, they won't use concurrent execution. Mostly the system uses concurrent execution
    • How are concurrent executions recorded in a notebook? Execution counter? Do different cells store which dependent kernel they use?
    • How does reactive execution work with this? For example, ipyflow (see paper here)

  • Community survey for kernels?

@Zsailer
Copy link
Member Author

Zsailer commented Feb 1, 2024

January 25th, 2024

Name affiliation GitHub username
Zach Sailer Apple Zsailer
William Stein CoCalc/SageMath @williamstein
Vidar T Fauske JP Morgan Chase vidartf
David Brochart QuantStack @davidbrochart
Johan mabille QuantStack @JohanMabille
Ian Thomas QuantStack @ianthomas23
Afshin T. Darian QuantStack @afshin
Jason Grout Databricks @jasongrout

  • Zach

    • Jupyter Server Representative nominations

  • Johan

    • Jupyter Kernels Representative nominations (ends February 1st)

  • Questions:

    • Is there a Jupyter project wide approach to documenting supported/deprecated Python versions (like NEP 29)?
    • If I write an extension today, can I target JupyterLab only or is there an expectation that I also support the classic notebook?
      • It depends on your audience.
      • There isn't an expectation to write extensions for both.
      • Notebook v7 makes it easier to write extensions that work in both.

  • [Ian] Update on concurrent execution? (requested by @jasongrout)

    • Next step: comments from community about what Ian has written
    • If no comments, update the JEP
    • @jasongrout and @Zsailer both have usecases that would benefit from concurrent execution

@Zsailer
Copy link
Member Author

Zsailer commented Feb 8, 2024

February 1st, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Michał Krassowski Quansight @krassowski
William Stein SageMath @williamstein
Ian Thomas QuantStack @ianthomas23
Kevin Bates Veritone @kevin-bates
David Brochart QuantStack @davidbrochart

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Feb 15, 2024

February 8th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Johan Mabille QuantStack @JohanMabille
Ian Thomas QuantStack @ianthomas23
Steve Silvester MongoDB @blink103
David Brochart QuantStack @davidbrochart
Mike Krassowski Quansight @krassowski

Agenda

Question:

@Zsailer
Copy link
Member Author

Zsailer commented Feb 22, 2024

February 15th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Ian Thomas QuantStack @ianthomas23
Jason Grout Databricks @jasongrout

@Zsailer
Copy link
Member Author

Zsailer commented Feb 29, 2024

February 22nd, 2024

Name affiliation GitHub username
Jason Grout Databricks @jasongrout
Sergey Kukhtichev IBM @skukhtichev
David Brochart QuantStack @davidbrochart
Johan Mabille QuantStack @JohanMabille
Ian Thomas QuantStack @ianthomas23

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Feb 29, 2024

February 29th, 2024

Name affiliation GitHub username
Johan Mabille QuantStack @JohanMabille
Vidar T Fauske JP Morgan Chase @vidartf
Zach Sailer Apple @Zsailer
Steve Silvester MongoDB @blink1073
Ian Thomas QuantStack @ianthomas23

Agenda

  • [Steve] Async Updates across the stack

    • Merged Ian's work to remove control queue from IPykernel
    • Combine this with David's work to remove Tornado IOLoop
    • Steve is working on grafting these two things together, which is proving to be challenging.
    • Zach and Steve are working on getting a new implementation of Async Kernel Manager out to remove ZMQ Stream and Tornado IOLoop.

  • [Nick] Schema repo discussion

    • Let's not do any RST for documentation.
    • Markdown forward.
    • That's aim for tooling that works both locally and on CI.
    • Get rid of Makefile
    • Add sphinx autobuild
    • Validate URI of schemas against their location in the repo
    • We should spend some time thinking about URI structure
      • Namespacing shouldn't be a question of "which project owns this schema", such as /jupyter_server/ or jupyter_client
      • Intead, let's namespace based on nouns, e.g. /server/, /kernels/, etc.

@Zsailer
Copy link
Member Author

Zsailer commented Mar 7, 2024

March 7th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Jason Grout Databricks @jasongrout
David Brochart QuantStack @davidbrochart
Ian Thomas QuantStack @ianthomas23
Steve Silvester MongoDB @blink1073

Agenda

  • [Steve] Releases and security updates
    • PR merged in server for enhanced security around release actions
    • Using a Github App for publishing. No longer using Github Admin Token. This also solves the issue of having to regularly update the token.
    • Can't use this for NPM yet.
    • Private release feature in jupyter-releaser (thanks Fred)
      • Useful for security patch releases; using a two part
        1. Publish to PyPI and create a tag.
        2. (1 week later) publish changelong and Github release using a separate workflow
  • [@jasongrout] Linux Foundation proposal
    • There is an open issue for discussion.
  • [@jasongrout] What is the current status of the prototype for concurrent kernel execution?
    • We might experiment with this at Databricks next week in an engineer hackathon
    • The current prototype is definitely demo code. You can create subshells, you can send execute requests. Potentially the status info is off. The main thread is a router instead of the main execution thread, where it needs to be. You can't delete subshells yet.
    • Anyio prototype is also in progress, and could impact the subshell work. David was working on this, but it has passed over to Steve now. This would make ipykernel asyncio-first, and would simplify future developments. Advantages: support for trio and uvloop.
    • related: AnyIO support in pyzmq
      FEAT: support AnyIO zeromq/pyzmq#1827

@Zsailer
Copy link
Member Author

Zsailer commented Mar 21, 2024

March 14th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Ian Thomas QuantStack @ianthomas23

Agenda

  • [Zach]
    • General announcements for Jupyter Server Team
      • Please read, review, comment (if you'd like) on the move to Linux Foundation proposalLinux Foundation proposal
      • Voting has opened for the Executive Council

@Zsailer
Copy link
Member Author

Zsailer commented Mar 28, 2024

March 21st, 2024

Name affiliation GitHub username
Jason Grout Databricks @jasongrout
David Brochart QuantStack @davidbrochart
Ian Thomas QuantStack @ianthomas23
Omar Jarjur Google @ojarjur
Zach Sailer Apple @Zsailer
Johan Mabille QuantStack @JohanMabille

Agenda

  • [Ryan, Gabriel, Jason] Lessons from exploring implementing the subshell demo in Databricks

    • Overall, the demo worked great!
    • Parent headers are per channel (e.g., "control", "shell"), so are inaccurate for subshells. We hacked a parent header per subshell
    • Recent changes for output per thread and eliminating the control queue conflict with the subshell prototype
    • We didn't finish swapping the main thread to be code execution instead of message routing, so interruption still didn't work
    • We added a shell_id parameter to create_subshell_request so we could explicitly set a subshell id (and thereby consolidate subshell ids if desired)
    • We changed the name of the thread to be shell-{name} to aid in debugging
    • Is there a reason we are using zmq/pickling instead of a Python Queue or SimpleQueue to communicate messages between threads?
    • Thread safety issues are tricky (like autocomplete requests might run 3rd party library code that assumes single-threaded)
    • Read up on subinterpreters coming in Python 3.13. Perhaps spawning a subinterpreter is a different kind of subshell, or perhaps subinterpreter is well-suited to a dependent kernel approach

  • [Omar] Requesting feedback on Navigating in the JupyterLab UI can prevent idle kernels from being culled. jupyter_server#1360

Reminder:

  • EC council election is currently going on, see your email if you are on a Jupyter council or committee
  • Linux Foundation proposal is currently under discussion

@Zsailer
Copy link
Member Author

Zsailer commented Apr 11, 2024

March 28th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
David Brochart QuantStack @davidbrochart
Piyush Jain AWS @3coins

Agenda

  • Zach

    • I can't host contributing hour today.

  • Ryan

    • debugger not working on background thread

  • Piyush

  • This issue was reported on gitter a few weeks back. Are there any suggestions on how to fix this.

    • Discussed the issue and the next step is to open an issue with all the information. Zach mentioned that this could be a regression from server 1.0, so should try with that. There is also an env variable JUPYTER_SERVER_ROOT which is not configurable at the moment, but could be made into a configurable, which can help nudge the CWD for terminals at runtime.

Bug around inaccurate execution_state in the client
- jupyter-server/jupyter_server#1395
- jupyter-server/jupyter_server#990

@Zsailer
Copy link
Member Author

Zsailer commented Apr 11, 2024

April 4th, 2024

Name affiliation GitHub username
Ian Thomas QuantStack @ianthomas23
R Ely Bloomberg @ohrely
Jason Grout Databricks @jasongrout
Maico Timmerman Adyen @MaicoTimmerman

Agenda

  • Ely:

  • Darian:

  • Jason: update on async kernel execution?

    • Ian may be able to start working on this a few weeks from now
    • Jason may be able to find some engineering time as well to coordinate with Ian on writing an implementation in May/June.
    • There are concerns about the tests that are broken with the anyio changes.
    • No one is currently actively working on the any-io test failures, but Ian has worked on at least one.

  • Maico: I'm looking for some support on an MR on the enterprise-gateway. I was hoping to utilize the meeting to get in touch with the correct person.

@Zsailer
Copy link
Member Author

Zsailer commented Apr 25, 2024

April 11th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Jason Grout Databricks @jasongrout
David Brochart QuantStack @davidbrochart
Afshin T. Darian QuantStack @afshin
Johan Mabille QuantStack @JohanMabille
Ian Thomas QuantStack @ianthomas23
Mike Krassowski Quansight @krassowski
Steve Silvester MongoDB @blink1073

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Apr 25, 2024

April 18th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Steve Silvester MongoDB @blink1073
Johan Mabille QuantStack @JohanMabille
Ian Thomas QuantStack @ianthomas23
David Brochart QuantStack @davidbrochart
Vidar Fauske JP Morgan Chase @vidartf
Mike Krassowski Quansight @krassowski

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented May 2, 2024

April 25th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
David Brochart QuantStack @davidbrochart
Vidar Fauske JP Morgan Chase @vidartf
Afshin T. Darian QuantStack @afshin

Agenda

  • Zach
    • SSC Rep moving into voting phase today
      • Multiple candidates
      • Rank-based voting
      • Single transferable vote strategy for counting votes.
      • Apache STeVe software we'll use to run the vote.
    • Async start_extension hook: Add async start hook to ExtensionApp API jupyter_server#1417
    • I won't be able to run Contributing Hour anymore (at least for the time being). I'll remove the calendar invite.

@Zsailer
Copy link
Member Author

Zsailer commented May 9, 2024

May 2nd, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Steve Silvester MongoDB @blink1073
David Brochart QuantStack @davidbrochart
Luciano Resende Apple @lresende
Ian Thomas QuantStack @ianthomas23

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented May 9, 2024

May 9th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Mike Krassowski Quansight @krassowski
Steve Silvester MongoDB @blink1073
Ian Thomas QuantStack @ianthomas23
A T Darian QuantStack @afshin

Agenda

  • Zach

    • Announcing new Jupyter Server SSC Representative!
      • Congratulations to Vidar Fauske for being elected as the new Jupyter Server SSC Repo
      • How do we make election results more transparent?
      • Add an SSC Drive to host election forms
        • Zach will open an issue on governance?
    • Volunteers to run the Jupyter Server/Kernels weekly call?

  • Mike

    • restoring full state (related to server side execution)
      • https://github.com/datalayer/jupyter-server-nbmodel
      • three missing things:
        • removing pending execution state indicator and replacing with execution count,
        • stdin boxes
        • timing metadata
      • discussions: Notebook cell execution jupyter_ydoc#169
      • Mike asked Zach's opinion on if REST API proposal listed above is the "right" way to go?
        • Zach said this proposal was motivated by some issues he saw where we leak resources/websockets too easily in Jupyter Server
        • He was aiming to simplify the API by providing a REST API for kernel messages and leverage the Event APIs as the single source of structured/schematized websockets.
        • Mike pointed out that one challenge of this approach is that, because the request (via REST) and the reply (events websocket) are not going through the same API, the cache needed to track their relationship might be difficult to implement in a way that doesn't cause memory leaks.
      • Discussion about what model should live on the server?
        • Idea from Nick. Instead of a single document model; have a "workspace" CRDT model.
        • Make the ydoc changes looks something like JupyterLab commands.
        • These in-memory models are really "UI models", telling each client how to rebuild the UI exactly how the user left it.
        • Imagine CRDT in Voila; RTC on widgets in voila would be amazing.

@Zsailer
Copy link
Member Author

Zsailer commented May 23, 2024

May 16th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Frederic Collonval WebScIT / Datalayer @fcollonval
Johan Mabille QuantStack @JohanMabille
David Brochart QuantStack @davidbrochart
Ian Thomas QuantStack @ianthomas23

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented May 23, 2024

May 23rd, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Johan Mabille QuantStack @JohanMabille
Ian Thomas QuantStack @ianthomas23
Piyush Jain AWS @3coins
Steve Silvester MongoDB @blink1073
Afshin T. Darian QuantStack @afshin
Frederic Collonval WebScIT / Datalayer @fcollonval

Agenda

  • Steve - recap of PyCon US 2024

    • Highly recommended talk from Anthony Shaw Unlocking the Parallel Universe: Sub Interpreters and Free-Threading in Python 3.13
    • Packaging Summit - we're working toward a Packaging working group, along with a dedicated effort at the PSF level including a UX expert for "onboarding onto Python", which may include a "blessed" workflow, documentation, new tool(s), etc.
    • Pittsburgh is great - try to go next year! - Try the Primanti Sandwich
    • The next location will be Long Beach, CA in 2026-27.

  • Ian - subshells (JEP91) real implementation in progress

    • Aiming for more accessible demo (e.g. via binder) for people to play with.

  • Discussion around Jupyter extension long-term support

    • Looked at CNCF's incubation project for guidance
    • Jupyter has less people resources
    • https://github.com/jupyter/docker-stacks
      • A great way to show opinion combinations of Jupyter pieces together and tests that they work together

  • Frederic

    • Discussion to collaborate on an extension implementing a kernel state machine on the server side; final goal would be to upstream it when stable.
    • Zach will open two lines of work
      • One on Jupyter Client with Async kernel manager
      • A new repo with a server extension and customer kernel manager with a kernel state machine

@Zsailer
Copy link
Member Author

Zsailer commented Jun 6, 2024

May 30th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Piyush Jain AWS @3coins
David Brochart QuantStack @davidbrochart
Zach Sailer Apple @Zsailer

Agenda

  • Zach
    • Next generation kernels API
      • Problems I'm trying to solve:
        1. the control and shell channel states are conflated.
        2. the kernel client on the server doesn't "know" the state of the kernel. For example, if the client is disconnected when the kernel state changes, recovering the true state isn't easy.
        3. every kernel websocket opens a new set of ZMQ sockets. This presents multiple issues, e.g. 1) ZMQ sockets are resource limited 2) it's difficult to track all of the places where messages are flowing (making (3) difficult to get right) 3) today, we're leaking sockets somewhere (Zach has seen this for long running servers, but hasn't tracked the cause)
      • Proposal to fix
        • (1) include the parent channel in the iopub messages, or track these messages server-side to distinguish status messages from shell/control.
        • (2) track two types of kernel states in the server (via KernelManagers)
          • lifecycle state, (e.g. starting, started, connecting, connected, terminating, terminated)
          • execution state, (busy, idle, unknown, dead)
          • Add a REST API (?) for fetching kernel state anytime the client needs to confirm or reconnect.
          • Use the event system (?) to emit kernel state?
        • (3) in jupyter-server, we should use a bit more discipline. Instead of opening individual sockets through the kernel manager API, we should open a single kernel client server-side. This opens a fixed set of ZMQ channels. All client->server->kernel connections, e.g. kernel websockets, should use this single kernel client.
          • Also, stop nudging the kernel for state. Just ask the server using the kernel state tracking from (2).

@Zsailer
Copy link
Member Author

Zsailer commented Jun 6, 2024

June 6th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
David Brochart QuantStack @davidbrochart
Steve Silvester MongoDB @blink1073
Zach Sailer Apple @Zsailer
Johan Mabille QuantStack
Omar Jarjur Google @ojarjur

Agenda

  • Zach

    • Follow up on "next-generation" kernel API
      • work in progress: https://github.com/Zsailer/nexgen-kernel-manager
      • Improvements made:
        • tracks kernel lifecycle state and execution state server-side.
        • uses a single kernel client (thus, single set of ZMQ channels) to communicate with the kernel. No need to open ZMQ sockets outside of this client.
        • uses a completely native asyncio approach to poll messages from the kernel, dropping the tornado IOLoop and ZMQStream logic.
        • simplifies the websocket connection logic
          • removes all nudging logic in the websocket handler, since the kernel manager owns this now.
          • the WS handle registers itself as a listener on the kernel client
          • the websocket can connect, even if the kernel is busy. (I think) this eliminates the necessity for "pending" kernels. Every kernel can be in a "pending" state.
      • how does this affect Omar's PR, which is trying to get an accurate execution_state on the server: Improve the busy/idle execution state tracking for kernels. jupyter_server#1429
        • We should proceed with reviewing this PR, if it doesn't increase our API service.
        • We should be able to simplify this PR but just "watching" the shell channel status messages.
        • Can we assume shell messages get queued?
          • If no, we might need to additional tracking of parent message ID
          • If yes, just tracking control channel should be enough.

  • Steve

@3coins
Copy link

3coins commented Jun 20, 2024

June 13th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Steve Silvester MongoDB @blink1073
Zach Sailer Apple @Zsailer
David Brochart QuantStack @davidbrochart
Johan Mabille QuantStack
Ian Thomas QuantStack @ianthomas23

Agenda

@3coins
Copy link

3coins commented Jun 22, 2024

June 20th, 2024

Name affiliation GitHub username
Piyush Jain AWS @3coins
Vidar T Fauske JP Morgan Chase @vidartf
Zach Sailer Apple @Zsailer
Andrii Ieroshenko AWS @andrii-i
Afshin T. Darian @QuantStack @afshin
Mike Krassowski Quansight @krassowski

Agenda

  • Demo of Parameterized Kernel Specs JEP

    • Context:

    • Vidar - Needs a flag to turn the save/load from notebook feature off/on, could cause shell injection attacks.

    • Zach - Separate the saving of kernel parameters in notebook to a separate kernel spec file, provide a save option.

      • Should the save kernel spec be a separate feature?
      • I don't think embedding detailed info about kernel runtime/environment in the notebook document is good broad solution. The moment that the notebook document leaves the current server, e.g. shared with a colleague, those parameters are essentially useless. You have to send the kernel + kernelspec.
      • In my experience doing something similar, it was better to save these details as a static (i.e. not parameterized) kernelspec.

  • Mike

    • securing config
      • A config that user can't read, example: imagine a billing document that can be saved in settings, but should not be accessed by the user.
        • Possible solution: Make file readable only by the server user in OS, would require elevating privileges of Jupyter server, which might not be ideal.
        • Alternate solution: Running extensions in a separate container than the main server, proxying the calls.
        • Nick/Zach: Comprehensive solution to isolate access with capability based model.

  • Nick

    • Demo of jupyter-speedscope extension to work with the speedscope profiler in the notebook

@Zsailer
Copy link
Member Author

Zsailer commented Jul 11, 2024

June 27th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Steve Silvester MongoDB @blink1073
Piyush Jain AWS @3coins
Vidar T Fauske JPMorganChase @vidartf
Jason Grout Databricks @jasongrout
Ian Thomas QuantStack @ianthomas23

Agenda

  • Release ipykernel live using Jupyter releaser
  • Briefly discussed sending contents over kernel message/comms
    • No one in this Jupyter Server call has worked on this problem at all.
  • Briefly discuss the subshell ipykernel branch
    • Doesn't break any new unit tests in ipykernel, though there are lots of tests that are broken, presumably because of anyio changes
    • Options for releasing ipykernel 7.x
      • have a long beta cycle where we finish and polish the subshell feature
        • may take until the end of the year or longer
      • (recommended) fix anyio breakages, release 7.0 with subshells as a beta/preview feature, and we keep iterating on subshells
        • Since subshells is backwards compatible, we should be able to do this, and it keeps up our development and release momentum
        • To make an ipykernel 7.0 release, then, we need to clean up the anyio CI breakages

@Zsailer
Copy link
Member Author

Zsailer commented Jul 11, 2024 •
edited
Loading

July 11th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Zach Sailer Apple @Zsailer
Steve Silvester MongoDB @blink1073
Piyush Jain AWS @3coins
Ian Thomas QuantStack @ianthomas23
Gonzalo Gasca Meza Google @gogasca
Johan Mabille QuantStack @JohanMabille
Omar Jarjur Google @ojarjur

Agenda

  • Vidar: Vote for Linux Foundation is approaching deadline. If you have any input (and are not directly represented yourself, please let me know you opinion either in private or in the discussions so that it can be taken into account): vote; dicsussion. The vote closes on Monday July 15, 2024.

  • Gonzalo: Feedback on: Pass session_id during Websocket connect jupyter_server#1440

    • pass the session_id from the websocket connection to the gateway.
    • Note: this is different than session_id from the Sessions REST API/service. The session service maintains a mapping of document to kernel with a session_id. The session_id mentioned in this issue, however, is a unique ID mapping a specific websocket connection to a kernel connection (i.e. jupyter_client) session.
    • Zach - is there logic in enterprise_gateway and kernel_gateway to handle this websocket connection session_id? I just did a quick Github search of the codebase and couldn't find anything. Maybe it would be good to open issues on both those repos to take advantage of this parameter.

  • Johan: Update of KernelSpec JEP: Parameterized kernel specs proposal jupyter/enhancement-proposals#87

    • Split the JEP, one for parameters, one for saving (or not) the parematers
    • Updated the secure kernelspec detection algorithm

  • Omar: Feedback on: Improve the busy/idle execution state tracking for kernels. jupyter_server#1429

    • As part of the review, maintaining a list of messages that trigger activity might get difficult.
    • Maybe maintaining a list of messages that shouldn't trigger activity would be better.
    • Zach - I'll work with Omar to develop a "block" list.

  • Steve: thoughts on [Proposal] Rename to hatch-jupyter jupyterlab/hatch-jupyter-builder#143 - rename to hatch-jupyter?

    • Concern is that folks relying on the old name don't get new releases of hatch-jupyter-builder.
    • We could make one last release of the "old" package that depends on the new one (unpinned). Folks can keep using the old name but it's using the new one under the hood going forward. Add deprecation warnings to hope that people see this change.
    • Zach - I added a comment to issue.

@Zsailer Zsailer mentioned this issue Jul 11, 2024
Closed
@Zsailer
Copy link
Member Author

Zsailer commented Jul 18, 2024

July 18th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Ian Thomas QuantStack @ianthomas23
Vidar T Fauske JPMorgan Chase @vidartf
Mike Krassowski Quansight @krassowski
Omar Jarjur Google @ojarjur

Agenda

@3coins
Copy link

3coins commented Jul 25, 2024

July 25th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Piyush Jain AWS @3coins

Agenda

  • Vidar: Reent repo activity: PR 1444

@Zsailer
Copy link
Member Author

Zsailer commented Aug 8, 2024

August 1st, 2024

Name affiliation GitHub username
Zach Sailer Apple Zsailer
Ian Thomas QuantStack @ianthomas23
Anastasiia Sliusar QuantStack AnastasiaSliusar

Agenda

  • We had low attendance today and no meeting agenda items, so we cancelled today's meeting.
  • The summer vacation time means slow days here :)

@Zsailer
Copy link
Member Author

Zsailer commented Aug 15, 2024

August 8th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Mike Krassowski Quansight @krassowski
Ian Thomas QuantStack @ianthomas23
Omar Jarjur Google @ojarjur

Agenda

  • Mike
    thoughts on transferring jupyter-ydoc Transferring jupyter ydoc (and pycrdt?) jupyterlab/frontends-team-compass#257

  • Ian

    • Been doing some maintenance on ipykernel
    • Intend to attend SSC meeting next week and request vote on JEP91 (kernel subshells) Add JEP for sub-shells jupyter/enhancement-proposals#91
    • As discussed a few weeks ago, planning ipykernel 7 release including anyio and subshells later in the year. Would be good to share some of the maintenance/reviewing workload for that release.

  • Omar

    • Thoughts/feedback on Add the ability to dynamically capture server logs from a running server. jupyter_server#1450 ?
    • Build an extension or core endpoint to dynamically change the log level and fetch the logs from the JupyterLab UI.
    • Comments from the group:
      • You could use the Python's logging config to emit a different level to a file or something similar.
        • You can append two logging emitters, set with two different levels, and dynamically listen to the streams.
      • Building a rich log viewer takes whole engineering team, so this might be more work than it appears.
      • Operators might not want users to see logs
        • There is an authorization layer in jupyter server that could turn this off by default.

@Zsailer
Copy link
Member Author

Zsailer commented Aug 15, 2024

August 15th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Ian Thomas QuantStack @ianthomas23
Vidar T Fauske JP Morgan Chase @vidartf

Agenda

The agenda was empty, so we had a few spontaneous conversations:

  • "Commenting" in Notebooks.

  • Jupyter-server and jupyverse

    • Today, there is no compatibility layer to make a server-side extension work in both Jupyter-server and jupyverse.
    • Extension authors shouldn't have to write both backends, so this needs to happen soon before consider giving users an option on which backend they'll use.

@Zsailer
Copy link
Member Author

Zsailer commented Aug 22, 2024

August 22nd, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Mike Krassowski Quansight @krassowski
David Brochart QuantStack @davidbrochart

Agenda

Mike:

Conversations after agenda
- Topic: Building a compatibility layer between Jupyter Server and Jupyverse
- Create a bridge library with a set of server "commands" (analogous to JupyterLab commands)
- Define commands that each service supports.
- Services speak commands, instead of calling live objects from other services.
- e.g. today, sessionsmanager calls the mapping kernel manager directly to start a kernel.
- Provide a way to register a "start_kernel" command. Whatever happens underneath this command doesn't matter to the caller.
- e.g. SessionManager just calls the start_kernel command. It doesn't need access to the live object.
- This bridge/command library could be used by jupyter_server, jupyverse, and extensions.

@Zsailer
Copy link
Member Author

Zsailer commented Sep 5, 2024

August 29th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Mike Krassowski Quansight @krassowski
David Brochart QuantStack @davidbrochart
Martin RENOU QuantStack @martinRenou

Agenda

Extending the jupyter-server's ContentsManager to enable new file formats

  • Could the base ContentsManager register custom file types?
  • Today, we set the type of content using two keys in the contents REST API model, type and format.
  • Extending would mean adding new type/format combinations.
  • Problems we observed:
    • The contents handler in the Jupyter Server owns the logic to check these keys, not the manager.
    • There is discrepancy between the formats allowed/not allowed in the Handlers and the contents manager. For example, "json" is not a valid format in the handler, but it is valid in the contents manager.
    • there is a lot of logic in the handler that should be pushed down into the manager.
    • Contents manager throws HTTP Errors? that's strange (and probably wrong)
  • In general, it would be ideal to have a registration/plugin point to bring your own file types to Jupyter Server without requiring folks to bring a custom contents manager to support them. Instead, we want to use the default contents manager, but we can act on files that aren't notebooks or text or base64 strings.
  • Comments in the chat:
    • The limitation of the contents GET approach will always be that it does not support streaming the content. Unless we have any ideas
    • maybe the "adapter" could have a public serialize() method and in a future an async aserialize() if we ever support streaming?
    • Compressed notebooks is also an idea

@Zsailer
Copy link
Member Author

Zsailer commented Sep 5, 2024

September 5th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
David Brochart QuantStack @davidbrochart
Omar Jarjur Google
Ian Thomas QuantStack @ianthomas23

Agenda

  • Curious what folks think about this PR, removing .so from the hidden list on ContentsManager.

  • Discussed a new proposed abstraction in JupyterLab, "ContentsProvider".

    • Add content providers jupyterlab/jupyterlab#16744
    • Here, a contents provider means the protocol through which contents data is coming into an IDrive.
    • This PR introduces a ContentsProviderRegistry with a default RESTContentProvider that talks to the Jupyter Server REST API.
    • Then, Jupyter Collaboration can introduce a new RTCContentsProvider that serves a collaborative document.
    • This allows us to drop the path prefix, "RTC:", from jupyter collaboration.

  • Python 3.13 is coming. We might see some things popping up in our tests. Just be prepared.

@3coins
Copy link

3coins commented Sep 19, 2024

September 12th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Piyush Jain AWS 3coins
Omar Jarjur Google @ojarjur
David Brochart QuantStack @davidbrochart
Afshin T. Darian QuantStack @afshin
Ian Thomas QuantStack @ianthomas23

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Oct 3, 2024

September 19th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Omar Jarjur Google @ojarjur

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Oct 3, 2024

September 26th, 2024

Name affiliation GitHub username
Omar Jarjur Google @ojarjur
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf

Agenda

Vidar:

@Zsailer
Copy link
Member Author

Zsailer commented Oct 10, 2024

October 3rd, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Omar Jarjur Google @ojarjur

Agenda

  • distutils is going away in Python 3.13. We need to update anywhere we use it.

    • Example that stood out, nbclassic
    • Open an issue on NBClassic. Support Python 3.13.

  • We had a fun, high level discussion/brainstorm about system architectures for Jupyter collaborative

    • We used the example of a single Jupyter widget poll for 800 students in a classroom.

@Zsailer
Copy link
Member Author

Zsailer commented Oct 10, 2024

October 10th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Ian Thomas QuantStack @ianthomas23
Omar Jarjur Google @ojarjur

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Oct 17, 2024

October 17th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Vidar T Fauske JP Morgan Chase @vidartf
Omar Jarjur Google @ojarjur

Agenda

  • Question about multi-tenancy in Jupyter Server
    • Should we flow the user model through the managers?
    • Flowing the user model is not the way to do this.
      • We don't want this token passed around through all managers.
      • This doesn't bring any value to the local user.
      • Peppering this everywhere in the server will likely break things.
    • Instead, flow a e.g. "blessed metadata" field that is backed by a spec. This allows us to pass things around for specific deployments in a generic way.
    • Is contextvars a solution?
    • We shouldn't be passing the tornado request handler in the contextvars
    • For testing multiple users: https://locust.io/

@Zsailer
Copy link
Member Author

Zsailer commented Oct 24, 2024

October 24th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Ian Thomas QuantStack @ianthomas23
A. T. Darian QuantStack @afshin
Vidar T Fauske JP Morgan Chase @vidartf
Omar Jarjur Google @ojarjur

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Nov 7, 2024

October 31st, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Ian Thomas QuantStack @ianthomas23
Vidar T Fauske JP Morgan Chase @vidartf
Omar Jarjur Google @ojarjur

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Nov 7, 2024

November 7th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Luciano Resende Apple @lresende
Zach Sailer Apple @Zsailer
Afshin T. Darian QuantStack @afshin
Omar Jarjur Google @ojarjur

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Dec 5, 2024

November 14th, 2024

Name affiliation GitHub username
Omar Jarjur Google @ojarjur
Vidar T Fauske JP Morgan Chase @vidartf
Ian Thomas QuantStack @ianthomas23

Agenda

@Zsailer
Copy link
Member Author

Zsailer commented Dec 5, 2024

November 21th, 2024

Name affiliation GitHub username
Vidar T Fauske JP Morgan Chase @vidartf
Luciano Resende Apple @lresende

Agenda

  • Based on previous discussion, here is the metadata service issue #71 for discussion.

@Zsailer
Copy link
Member Author

Zsailer commented Dec 5, 2024

December 5th, 2024

Name affiliation GitHub username
Zach Sailer Apple @Zsailer
Omar Jarjur Google @ojarjur
Andrii Ieroshenko AWS @andrii-i
Vidar T Fauske JP Morgan Chase @vidartf
Ian Thomas QuantStack @ianthomas23

Agenda

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@3coins @Zsailer