Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

distributed_job decorator will not accept the shm_size (shared memory parameter) #100

Open
snavyareddy opened this issue May 8, 2024 · 0 comments

Comments

@snavyareddy
Copy link

@distributed_job(Framework.MPI, num_nodes=2, processes_per_node=1)

When using MPI framework and using more nodes in azureml the shared memory (shm_size) is only 2gb by default,
In my usecase, It is giving the memory issue.
image

I want to increase the shm_size, which was not possible with kedro azureml

But we can pass the shm_size, If we use normal azureml sdk v2 without kedro, while creating the job in the job schema
attaching the example job schema
image

If I want to have the same in kedro azureml, What is the way?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant