-
Notifications
You must be signed in to change notification settings - Fork 110
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run domains in a threadpool #1218
Comments
If we run against a setup that has a high number of tables, we can exceed the limit of threads and Readyset will panic:
A workaround is to limit the number of tables either via:
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Previously, we used tokio::task::block_in_place to run domains, which blocks the thread it's currently running on until the task completes. This prevents the executor from using that thread to make progress on any other tasks, which is not efficient. CL-1268 removed the
block_in_place
strategy by spawning a native OS thread for each domain instead, which is how the system works today. This is highly resource-inefficient and has also led to us hitting the upper limits on the number of threads with active tracing spans (4096).A quote from that CL:
Generally speaking, blocking I/O bound work is very well-suited to tokio's built-in blocking threadpool. Further, it is now possible to configure the size of the blocking threadpool using the max_blocking_threads method on the runtime builder. We should re-investigate the performance of tokio's
spawn_blocking
method in the context of domains.For work that is CPU-bound, we should consider using the rayon crate, which is typically the go-to tool for spawning blocking CPU-bound tasks.
The text was updated successfully, but these errors were encountered: