Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mlos_bench: if a result.___ value is -1 register trial as FAILED #800

Open
yshady opened this issue Jul 16, 2024 · 2 comments
Open

mlos_bench: if a result.___ value is -1 register trial as FAILED #800

yshady opened this issue Jul 16, 2024 · 2 comments

Comments

@yshady
Copy link

yshady commented Jul 16, 2024

sometimes benchbase will put -1 for latency throughput etc but we register the trial as succeeded

@bpkroth
Copy link
Contributor

bpkroth commented Jul 17, 2024

This is more an error in the client side benchbase output processing scripts than MLOS.
There are some related improvements for reporting success/failure of a trial, but I think the client side should be investigated first.

See Also:

@bpkroth
Copy link
Contributor

bpkroth commented Jul 17, 2024

To put it another way, there's not a great way for MLOS to know the semantics of what the "Latency (microseconds) 99th Percentile" metric from benchbase is.

I suppose one thing we could do is extend the objective_targets descriptor to include some "bounds" info (e.g., "min": 1, "max": null) and reject the trial if those aren't satisfied, but that would mostly just help in discovering the error in the benchbase scripts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants