Usernetes with bypass4netns on multi-node

[vsoch]

Hi @AkihiroSuda ! 👋

I want to introduce you to @lisejolicoeur, who has joined our team this summer (with @milroy) to work specifically on Usernetes networking! We are opening this issue to share our progress, and ask for your expertise for next steps to try.

Specifically, @lisejolicoeur has led us to extend our original setup on AWS to add back bypass4netns with multi-node, and we have taken notes that detail our approach. Lise is leading the work and is experienced with Kubernetes networking, and we are hopeful to start discussion here about next steps we might try. If we work together, we might be able to figure out solutions to some of the current challenges with getting bypass4netns working in this multi-node setup.

Our proposal is that if you might have time, please review our setup, and take note of the points of interest and questions indicated with ❓️. Please feel free to comment on or otherwise edit the document - it is a working space for us! We are at the point where we have a control plane running (without bypass) and two worker nodes (with bypass) and can shell into pods on other nodes, but cannot ping from pod to pod. Please let us know if you see anything that sticks out, and what we might try next.

Thank you, and happy weekend! 🥳

[AkihiroSuda]

I didn't read the entire note yet, but etcd is not needed for multi-node.
The acceleration works for Pod-to-Service, not for direct Pod-to-Pod though

[AkihiroSuda]

See also https://static.sched.com/hosted_files/containerplumbingdays2024/f1/Usernetes%20Gen2%20-%20Kubernetes%20in%20Rootless%20Docker%2C%20with%20Multiple%20Nodes.pdf

Especially slide 13