Worker-Scheduler SSL causing performance pain

[mrocklin]

I'm running a few experiments at moderate scale (100 4-core workers with many small tasks) and I'm finding that reading/writing on SSL sockets is the primary bottleneck. How can we handle this?

We could remove TLS security from workers/schedulers . This is currently redundant with our use of security groups. We've taken a belt-and-suspenders approach to network security so far, but maybe this performance issue encourages us to drop it, and just go with security groups. We would continue to use TLS/SSL between the client and scheduler.

[mrocklin]

dask/distributed#4443

[mrocklin]

I'm going to work with OSS folks to figure this out before bugging people here. Closing for now.