Evaluate Profile-Guided Optimization (PGO) and LLVM BOLT

[zamazan4ik]

Hi!

Recently I did a lot of PGO benchmarks on different software - all available results are here. According to my tests, PGO can help with achieving better performance in many workloads. Since Lychee cares about performance (I think so because the project has benchmarks), I think it would be a good idea to test PGO on it.

So I tested PGO on Lychee on my Macbook M1 Pro, macOS 13.4, Rust 1.72 and here are the results. All tests was done via cargo bench. PGO optimization is done with cargo-pgo.

Release:

extract from large docs time:   [12.814 ms 12.908 ms 13.045 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe

Release + PGO optimization (compared to Release), cargo pgo optimize bench command:

extract from large docs time:   [10.084 ms 10.098 ms 10.113 ms]
                        change: [-20.758% -20.030% -19.527%] (p = 0.00 < 0.05)
                        Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high severe

Release + PGO instrumented, compared to Release (I publish these numbers so you can estimate how Lychee is slow in the PGO instrumentation mode), cargo pgo bench command:

extract from large docs time:   [39.635 ms 39.781 ms 39.874 ms]
                        change: [+205.32% +207.99% +210.21%] (p = 0.00 < 0.05)
                        Performance has regressed.

As you see, at least in the provided by the projects benchmarks, PGO allows to achieve better performance.

I can suggest to do the following things:

• Add a note to the Lychee documentation about building with PGO. In this case, users and maintainers who build their own Lychee packages will be aware of PGO as an additional way to optimize Lychee
• Optimize provided by Lychee team binaries on the CI (like it's already done for other projects like Rustc), if you have any
• After that, I suggest trying to apply LLVM BOLT as an additional post-PGO step. Rustc already does it on some platforms.

[mre]

Apologies for the delay.
The numbers look promising.
At the end you mentioned Ruff, but I assume you meant lychee.
If you have the time, I'd be happy about the CI optimizations you mentioned.

[zamazan4ik]

Apologies for the delay.

No worries :)

At the end you mentioned Ruff, but I assume you meant lychee.

Whoops! Sorry - fixed that now.

If you have the time, I'd be happy about the CI optimizations you mentioned.

I will try my best but right now have quite a tight backlog with other projects. Hopefully, once will find enough time for Lychee.

[mre]

@zamazan4ik, is still something you'd like to tackle in the foreseeable future?

[zamazan4ik]

is still something you'd like to tackle in the foreseeable future?

Yep! I just want to finish my article about PGO. After that as a next step, I think I will get enough free time for integrating PGO into the build pipelines for more open-source projects (like Lychee).

[mre]

Awesomeness! Thanks for the update. 😊

[mre]

Uhm, that's not article, that's a book! 😆 You should actually consider to make this an ebook or so.

[zamazan4ik]

Uhm, that's not article, that's a book! 😆 You should actually consider to make this an ebook or so.

Huh! From my estimations, it's a half-finished article (or just a bit more like 60%). I am considering in the future creating something like a "PGO book" (as we already have an awesome Rust book, and many other similar ebooks). If the community will give feedback about creating a book - why not? Right now, you are the first person with such a request ;)

[mre]

Not an expert, but it looks like one of the most complete resources on the topic, especially with regards to Rust. An awesome book sounds like a great idea to me. It could make the content more accessible.

[mre]

Still open for PRs, but I'm closing this as it's technically not an issue with the project. Perhaps we can revisit this in the future.

[zamazan4ik]

Still open for PRs, but I'm closing this as it's technically not an issue with the project.

Then I suggest converting the issue into the Discussion (and moving to the "Ideas" section). It will be the better place for this activity since I agree with you - it's not a bug or something like that.

[mre]

Good idea!