Check benchmarks in CI

[brandonwillard]

We need an Actions/CI step that automatically compares benchmark results against current main and notifies us when there is a significant regression before a PR is merged.

We've used github-actions-benchmark in the past, but it seemed overly complicated, was more about serving a website with benchmark results (which is complexity we don't need right now), and wasn't actually able to perform the desired checks on each PR—at least not without some undesirable trade-offs.

If anyone knows of an existing workflow we can adapt, or alternative Action, please tell us!

N.B. https://labs.quansight.org/blog/2021/08/github-actions-benchmarks

[lapp0]

Good reading materials @brandonwillard

Here are some examples I found of ASV in use:

• dashboard of historical performance: https://asv-runner.github.io/asv-collection/pandas/
• PR comment in a repo with asv configured for PR actions: Reduced shared parquet file contention in testing lincc-frameworks/tape#440 (comment)
  • Corresponding asv-pr.yml: https://github.com/lincc-frameworks/tape/blob/8f33ee8a2a7ab77cbc337ebc0c121b9fdb17dba1/.github/workflows/asv-pr.yml

ASV seems like a great solution, it can be run in GitHub Actions while minimizing comparison error, is widely used by major python libraries, and appears to have relatively easy setup.

Happy to get started on this if you agree!

[rlouf]

Note that we should track both execution speed and memory usage.