Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bulk_register can fail when existing trials are all Failed and do not have result columns yet #831

Open
eujing opened this issue Aug 2, 2024 · 0 comments

Comments

@eujing
Copy link
Contributor

eujing commented Aug 2, 2024

If an experiment happens to fail on the first trial (maybe quota / provisioning issues) and then MLOS crashes after due to our user scripts expecting a result metric, trying to start the same experiment again gets pandas indexing errors for missing columns at bulk_register.

image

In such cases, _adjust_signs_df gets called with potentially empty data frames, or ones that do not contain the target opt columns yet because of failed runs.

Then this line will fail with indexing errors:
image

Maybe the dataframe can be created first before adjusting signs, and then checked if the target columns exist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant