Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: benchmark setup #183

Merged
merged 17 commits into from
Sep 26, 2024
Merged

feat: benchmark setup #183

merged 17 commits into from
Sep 26, 2024

Conversation

alexkeizer
Copy link
Collaborator

Description:

Add a basic benchmark setup:

  • benchmark command, which has a theorem-like syntax, but doesn't get added to the environment, and runs the proof 5 times, reporting both the individual and average runtime,
  • similarly, a logHeartbeats tactic combinator, which measures the heartbeats used by an arbitrary tactic.
  • Add a benchmark folder, with files for SHA512 runs with: 150, 225, and 400 steps (which are the numbers I've been reporting so far)
  • Adds a make benchmark Make-target, to build the aforementioned files. Crucially, this target is not part of make all, for all our sanity's sake (running 400 steps of SHA512 for 5 times takes a while)

Testing:

What tests have been run? Did make all succeed for your changes? Was
conformance testing successful on an Aarch64 machine? yes

License:

By submitting this pull request, I confirm that my contribution is
made under the terms of the Apache 2.0 license.

@alexkeizer alexkeizer requested a review from shigoel as a code owner September 23, 2024 22:26
@alexkeizer
Copy link
Collaborator Author

@shigoel, @bollu: do we want to run benchmarks in CI? It could be nice to gather data automatically, but I worry a bit that the data might be unreliable (since we have little to no control over what else is running on the CI worker), and that it might slow down CI too much (just doing the 400 step benchmark alone already takes 10 minutes)

Benchmarks/Command.lean Outdated Show resolved Hide resolved
Co-authored-by: Siddharth <[email protected]>
Benchmarks/Command.lean Outdated Show resolved Hide resolved
Co-authored-by: Siddharth <[email protected]>
@shigoel
Copy link
Collaborator

shigoel commented Sep 24, 2024

@alexkeizer I like the idea of benchmarking automatically in CI (even with all the caveats that we can't control what else is running on the worker). Maybe we can separate the benchmarking job from others so that folks don't have to wait for the benchmarking job to finish before hitting merge, etc. if they don't want to.

@shigoel
Copy link
Collaborator

shigoel commented Sep 24, 2024

@alexkeizer Also, could you please update the README to describe the new benchmark target?

@bollu
Copy link
Collaborator

bollu commented Sep 24, 2024

A separate build job makes sense to me, and PR LGTM!

@shigoel
Copy link
Collaborator

shigoel commented Sep 25, 2024

@alexkeizer FYI: incoming merge conflicts here.

@alexkeizer
Copy link
Collaborator Author

alexkeizer commented Sep 25, 2024

@shigoel I've merged main, added a line to the README, and made benchmarking part of the CI (on the ubuntu runner only, figured that'd be enough). This should now be ready to merge

Copy link
Collaborator

@shigoel shigoel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@shigoel shigoel merged commit 4414dc5 into leanprover:main Sep 26, 2024
3 checks passed
shigoel added a commit that referenced this pull request Sep 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants