Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

usernetes with efa on AWS #322

Open
vsoch opened this issue Mar 9, 2024 · 6 comments
Open

usernetes with efa on AWS #322

vsoch opened this issue Mar 9, 2024 · 6 comments
Labels
question Further information is requested

Comments

@vsoch
Copy link

vsoch commented Mar 9, 2024

hey @AkihiroSuda - happy weekend! I have a full setup working with flux and usernetes on AWS, and I added in the elastic fiber adapter (EFA) but it's absolutely not running. The link there has some background - it needs to leverage drivers on the host. Is there an extra bit of information / bind I need to add to the docker-compose setup to allow for that to happen? For context, the operator installs OK, and lammps even starts up OK, but (on a very small problem size that usually is done in 1-2 seconds on a bad machine) it's basically hanging:

$ kubectl  logs -f flux-sample-0-tjbkg -f
Defaulted container "flux-sample" out of: flux-sample, flux-view (init)
LAMMPS (7 Feb 2024 - Development - 8819275)

And the above hangs there. I'm thinking perhaps for usernetes I need to bind the driver location on the host to somewhere in usernetes? Or something else? Let me know if you have insights. As always, thank you for your help!

@AkihiroSuda AkihiroSuda added the question Further information is requested label Mar 9, 2024
@vsoch
Copy link
Author

vsoch commented Mar 9, 2024

oh wow it ran... but it was SO SLOW!

image

oh my! I do wonder if I'm missing a bind... actually I think it's there? EFA on the host I think is here:

$ ls /dev/infiniband/
uverbs0

and in the container I see it too:

# ls /dev/infiniband/uverbs0 
/dev/infiniband/uverbs0

But (even though efa is installed in the container) it could still be that it's not working... going to quickly run the tests.

@vsoch
Copy link
Author

vsoch commented Mar 9, 2024

Ah interesting - so the bind is to the docker-compose base images, but the containers don't see efa!
image

So there is some issue there. I wonder if installing the daemonset might work...

@vsoch
Copy link
Author

vsoch commented Mar 9, 2024

Ah, got it working! I will post the full update tomorrow - basically I needed to hack the daemonset a bit, and then add the correct annotations for it to bind to the pod (of the job). Then I could run a sleep job, shell in, install fi_info for libfabric, and see efa and run the tests.

It's super late here but I'm going to be doing experiments soon and can post the full details of the setup.

@vsoch
Copy link
Author

vsoch commented May 26, 2024

Update here - I think we've solved all the performance issues, but I need to do a few more scaled tests! @AkihiroSuda can I ask you a quick question? For your usernetes paper here:

https://arxiv.org/pdf/2402.00365v1

Does that component in Figure 1 (the "Intermediate NetNS" mentioned where the incoming / outgoing get routed through) get bypassed given using EFA? https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html

More specifically, in that diagram for EFA, would you say the usernetes "Intermediate NetNS" is part of the TCP/IP stack? Or the ENA device? https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa.html#efa-basics. I ask because I did some tweaks to our setup and was able to get the same performance on bare metal as in usernetes, and I think I can explain it based on those diagrams (if this is the case). Thanks for your help!

@AkihiroSuda
Copy link
Member

The "intermediate netns" just refers to RootlessKit's namespace i.e., the namespace of dockerd

@vsoch
Copy link
Author

vsoch commented May 26, 2024

So if we use the libfabric API that connects directly to the device (hardware) I'm guessing we are bypassing all of that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants