-
Notifications
You must be signed in to change notification settings - Fork 598
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Azure Batch: Add disk size to slots calculation #4920
Comments
how would you determine the available disk storage for a given VM / queue? |
I'm thinking way simpler than that. if |
I see, so if the user specifies a machine type or a queue with an implied machine type then we can use |
Correct. It's not perfect, but it might help catch a few mistakes. We would need to add the disk directive to these three places: nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy Line 565 in 735fb8b
nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy Lines 158 to 176 in 735fb8b
nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy Lines 158 to 176 in 54ad624
|
Better idea, we just turn the disk size into one of the compute slots. E.g., a job that requires 1 cpu, 1gb of memory and 128gb of storage on a machine with 16 cores, 64gb of memory and 256gb of storage would currently occupy 1 slot. If we update the system it will occupy 8/16 slots. See relevant code here: nextflow/plugins/nf-azure/src/main/nextflow/cloud/azure/batch/AzBatchService.groovy Lines 245 to 269 in c713ad5
|
Sounds good to me. Care to give it a go? 😄 You have everything you need from the |
Post summit. Maybe on the plane 😆 |
New feature
When using Azure Batch, Nextflow will reject a process if it has too many CPUs for the worker machine.
However, Azure Batch VMs come with a fixed disk and it's common that the Nextflow process runs out of storage. There are many, many issues about this on the Nextflow Slack! The typical workaround is to increase the number of CPUs an individual process requires, however it would be better to support the disk directive so we can directly enforce the VMs have the right sized disk.
Although we can't enforce it properly (i.e. make sure tasks are only assigned to a VM with enough space), being able to prevent users trying to run a task on a machine which is too small would catch some of the issues.
Usage scenario
When running on Azure Batch, raise an error if a task is assigned to a queue which does not contain sufficient storage.
Suggest implementation
The text was updated successfully, but these errors were encountered: