Slightly overhaul performance tests

[swissspidy]

This originally started as a proof-of-concept for running tests with object cache, but ended up covering quite a bunch more.

Example:

(View workflow run)

Specifically:

• Adds a test matrix to the GitHub Action to run performance tests both with and without an external object cache (Memcached).
  • Could even use this to test different PHP versions or so in the future.
• Consolidates duplicate/redundant test files in favor of using some for loops instead.
  • Makes the code more maintainable and allows more easily testing more combinations in the future (e.g. testing all default themes).
  • Not using the workflow matrix for this because adding new jobs would cause too much overhead and make the whole suite slower.
• Updates log-results.js so that the expected format stays the same, prevents the dashboard from breaking.
• Improves test setup and cleanup by disabling external HTTP requests and cron for the tests, as well as deleting expired transients and flushing the cache in-between. This should aid the test stability.
• When testing the previous commit / target branch, this now leverages the already built artifact from the build process workflow 😻 . This way, we avoid building WordPress twice and ensure that both the current commit & the target commit use the same script for running the tests.
• Upload raw test results as artifacts to easily inspect them locally.
• Tracks usage of external object cache, memory usage, and number of db queries in Server-Timing.
• Sends Server-Timing header on the admin so we get more insights there.
• Adds support for using repeatEach in Playwright. Now there are 2 repetitions with 20 iterations each.
• Generally overhauls compare-results.js to improve output. Adds standard deviation (STD) and median absolute deviation (MAD) as well.

Good to know:

• If the median wpDbQueries is 0 in the memcached results test, it means all db queries were served from the cache in subsequent tests. If the number is higher, than there are some uncached queries.
• If the number of db queries differs between repetitions and iterations, decimal numbers are to be expected.

---

Build/Test Tools: Overhaul performance tests to improve stability and cover more scenarios.

Simplifies the tests setup by leveraging a test matrix, improving maintenance and making it much easier to test more scenarios. With this change, tests are now also run with an external object cache (Memcached). Additional information such as memory usage and the number of database queries is now collected as well.

Improves test setup and cleanup by disabling external HTTP requests and cron for the tests, as well as deleting expired transients and flushing the cache in-between. This should aid the test stability.

When testing the previous commit / target branch, this now leverages the already built artifact from the build process workflow. Raw test results are now also uploaded as artifacts to aid debugging.

Props swissspidy, adamsilverstein, joemcgill, mukesh27, desrosj, youknowriad, flixos90.
Fixes #59900

---

Trac ticket: https://core.trac.wordpress.org/ticket/59900

---

This Pull Request is for code review only. Please keep all other discussion in the Trac ticket. Do not merge this Pull Request. See GitHub Pull Requests for Code Review in the Core Handbook for more details.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Core Committers: Use this line as a base for the props when committing in SVN:

Props swissspidy, adamsilverstein, joemcgill, mukesh27, desrosj, youknowriad.

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[adamsilverstein]

Amazing work!

[joemcgill]

Thanks @swissspidy, I really like these updates.

Do we need to make any updates to the https://www.codevitals.run/project/wordpress dashboard to be able to collect some of these new metrics (e.g., DB calls, additional themes, etc.) over time or adjust anything else about how these are now being collected?

I also think we may want to consider breaking this workflow into two separate workflows:
the performance test runner, and the main performance test workflow that defines the matrix and uses the runner to collect results, similar to how the phpunit-tests.yml and phpunit-tests-run.yml workflows are set up. This would make it easier for us to run manual workflows for a specific use case. However, this is a good iteration for now.

[mukeshpanchal27]

Thanks @swissspidy for the PR! Nice work. Left some questions.

[swissspidy]

Do we need to make any updates to the codevitals.run/project/wordpress dashboard to be able to collect some of these new metrics (e.g., DB calls, additional themes, etc.) over time or adjust anything else about how these are now being collected?

I updated the log script so that the existing metrics should be collected as usual. Now that we finally collect memory usage and server-timing in the admin, those fields, which were previously empty, can finally be displayed properly too.

For tracking things like db queries, changes to the dashboard are indeed necessary, but I usually handle this on Slack directly with Riad :)

That said, the dashboard really doesn't scale for that many metrics. So if we want to track all the default themes etc. then we need to improve/switch the dashboard first.

[youknowriad]

I agree that the UI of the dashboard is not meant to handle a huge list of metrics. I don't mind much If we think there are alternative dashboards that we can switch to without too much hassle.

If not, we can also implement this issue to have like "main metrics" and "all metrics" pages per project or something.

[desrosj]

The GHA related aspects look great. I added a few questions/suggestions.

[desrosj]

I also think we may want to consider breaking this workflow into two separate workflows:

@joemcgill I'm working on something like this in #6232 for all workflows because there are huge benefits for maintainability. Let's overhaul the workflow first, and then work on splitting it into a callable workflow there.

[swissspidy]

Committed in https://core.trac.wordpress.org/changeset/58076 🤞