[B] nf build tries to run a builder job of the preset that is not available at the moment

[eviltwinhans]

Summary

When only cpu-small presets are available and all cpu-medium presets are not available, neuro-flow build train still tries to run a builder job of cpu-medium preset condemning a user to possibly eternal expectation that this preset will be released.

Steps to reproduce

1. Setup: cluster with a small cpu node pool (e.g. Cluster Power) - 3 CPU, 13 GB in total, 3 cpu-small (1 CPU, 4 GB) or 1 cpu-medium (3 CPU, 12 GB).
2. 1 cpu-small job should run (not sure if a terminal job started by you will do the trick, issue was initially detected due to the job-38a589b5-d71f-4e81-8b20-62218a49c90f job that ran on k8s node and was invisible in CLI or UI)
3. Initiate a neuro-flow project
4. Run neuro-flow build train
5. Wait for a builder job to start for 10 minutes:

    INFO: Submitting a builder job
√ Job ID: job-8e945350-5ba2-4069-9a34-20b927cb90ea
- Status: pending Creating
- Status: pending Scheduling ◥ [644.1 sec]

Expected result

nf build should have started running builder job using the cpu-small preset (+ alert user that cpu-small is not the best preset for this cause) instead of eternally waiting for cpu-medium job to become available.

Environment

Mandatory:

• neuro-extras version: 22.2.2
• neuro CLI version: 22.6.2

Recommended:

• OS version: macOS 12.4 (21F79) Monterey