You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Secure clusters are nice. However, SSL reduces network performance, to some degree unavoidably, and potentially to a further degree because of issues in Tornado or dask (#5258 (comment), #4443).
In some cluster architectures, the scheduler might be exposed to the Internet, but workers are hidden behind an internal network. In those cases, you might want workers to communicate with each other (and possibly the scheduler) over unsecured TCP for data-transfer performance, while keeping scheduer<->client comms over secure TLS.
How could we do:
worker<->worker comms over TCP, everything touching the scheduler TLS
worker<->worker and worker<->scheduler comms over TCP, scheduler<->client over TLS
1 seems both easier and more of a performance win, but 2 might be interesting once we figure out 1.
If we make this change, would suggest we only use workarounds like this one and this one for secure connections and disable them otherwise (as they likely impact perf as well)
Though am curious to what extent this is an issue primarily with Tornado ( #5258 (comment) )
Secure clusters are nice. However, SSL reduces network performance, to some degree unavoidably, and potentially to a further degree because of issues in Tornado or dask (#5258 (comment), #4443).
In some cluster architectures, the scheduler might be exposed to the Internet, but workers are hidden behind an internal network. In those cases, you might want workers to communicate with each other (and possibly the scheduler) over unsecured TCP for data-transfer performance, while keeping scheduer<->client comms over secure TLS.
How could we do:
1 seems both easier and more of a performance win, but 2 might be interesting once we figure out 1.
cc @jcrist
The text was updated successfully, but these errors were encountered: