-
Notifications
You must be signed in to change notification settings - Fork 18
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: benchmark setup #183
Conversation
…utput more legible
@shigoel, @bollu: do we want to run benchmarks in CI? It could be nice to gather data automatically, but I worry a bit that the data might be unreliable (since we have little to no control over what else is running on the CI worker), and that it might slow down CI too much (just doing the 400 step benchmark alone already takes 10 minutes) |
Co-authored-by: Siddharth <[email protected]>
Co-authored-by: Siddharth <[email protected]>
@alexkeizer I like the idea of benchmarking automatically in CI (even with all the caveats that we can't control what else is running on the worker). Maybe we can separate the benchmarking job from others so that folks don't have to wait for the benchmarking job to finish before hitting merge, etc. if they don't want to. |
@alexkeizer Also, could you please update the README to describe the new |
A separate build job makes sense to me, and PR LGTM! |
@alexkeizer FYI: incoming merge conflicts here. |
…polates `state` as a variable
The name for `s0` was previously inaccessible, which tripped up the `inferStatePrefixAndNumber` warning
@shigoel I've merged main, added a line to the README, and made benchmarking part of the CI (on the ubuntu runner only, figured that'd be enough). This should now be ready to merge |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks!
Description:
Add a basic benchmark setup:
benchmark
command, which has a theorem-like syntax, but doesn't get added to the environment, and runs the proof 5 times, reporting both the individual and average runtime,logHeartbeats
tactic combinator, which measures the heartbeats used by an arbitrary tactic.make benchmark
Make-target, to build the aforementioned files. Crucially, this target is not part ofmake all
, for all our sanity's sake (running 400 steps of SHA512 for 5 times takes a while)Testing:
What tests have been run? Did
make all
succeed for your changes? Wasconformance testing successful on an Aarch64 machine? yes
License:
By submitting this pull request, I confirm that my contribution is
made under the terms of the Apache 2.0 license.