Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better heuristic for when to optimize target() metrics #3176

Open
Zac-HD opened this issue Dec 4, 2021 · 0 comments
Open

Better heuristic for when to optimize target() metrics #3176

Zac-HD opened this issue Dec 4, 2021 · 0 comments
Labels
internals Stuff that only Hypothesis devs should ever see performance go faster! use less memory!

Comments

@Zac-HD
Copy link
Member

Zac-HD commented Dec 4, 2021

Currently, we run all of our optimizer routines exactly once, starting at (or just after) half of our max_examples budget. This works pretty well for the default max_examples=100, but I've noticed increasing use of Hypothesis with very large example budgets (and target()) for one-off searches for some example - and we can do much better for this workflow. Specifically, we should:

  • better support interleaving of ordinary generation with targeted optimization; e.g. "run an optimize step, then generate that many examples, then the next optimize step...". This still aims to spend up-to-half our total examples optimizing.
  • pick a smoother heuristic for when-to-optimize. Initial proposal: maintain current behaviour for max_examples < 1000; for max_examples >= 1000 use the incremental mixture instead, starting after 200 examples generated by our usual means.
  • for long runs, we might finish running the optimizer. How should we decide to start a second full pass? We don't want unproductive re-runs (as may be implicated in Pareto-optimizer sometimes causes tests to run much more slowly #2985), but an additional run must sometimes make sense...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
internals Stuff that only Hypothesis devs should ever see performance go faster! use less memory!
Projects
None yet
Development

No branches or pull requests

1 participant