Reading generated sources is extremely slow #94
Comments
|
I completely understand, this is 100% valid criticism. Part of it is sphinx: more rst docs -> more work. Some users reported that using PyPy rather than Cython was enough of a speed boost to make it worth it. The other part is more difficult, when my blackout ends (work is borderline killing me these days) the first thing we need to do is rewrite breathe. I think we pinpointed it to the caching strategy being the problem, I think the majority of the time is spent looking up things from the cache (breathe caches the xml and when it's getting rendered looks it up from the cache). It seems I forgot to enable parallel builds too, I will try and add that today since it's a small change, please nudge me if I forget. But right now exhale may only be appropriate for small to medium projects. I know the pytorch docs take ~30 minutes to build as well, they just setup a cron build to do it every so often rather than per commit I think. You're only other option (which isn't ideal) is to not document as many items... |
|
(the sphinx part: to test, try using alabaster theme which is lightweight, some html themes are much more expensive than others) |
|
Thanks for your prompt answer, i'll give it a shot using another theme. |
|
Here is the error I get: Here is the content of the faulty file: It seems it fails on function macros. Any way to your knowledge to workaround this ? |
|
Lol, it's kind of funny that it's an Ok I've got some time set aside tonight to put some band-aids on exhale and I'll see if there's anything clever I can do about macros. I'm not very optimistic there's much I can do, will create a minimal repro but I expect that the doxygen XML is misleading exhale. To at least get past the macro I suggest skipping it from being documented at least in it's current form. I can't find the config right now but I remember there being something special about function macros that you can set on the doxygen side, I need to add some test cases for that. Another alternative is to predefine as shown with the dllexport example here https://www.doxygen.nl/manual/preprocessing.html which should result in it getting removed (?) Not exactly a fix, but should enable the build to at least succeed. To clarify the breathe side, I think it all stems from memory leaks in minidom which crushes large projects, but it will be a lot of work to redo. So if (a) PyPy and (b) me (soon) adding parallel do not reduce build times enough then it probably isn't going to be a good choice for your project at this time. It is strange that the doxygen index version is not affected though, and may indicate we have other options. Exhale should be reporting timing metrics near the beginning of the sphinx build log about how long it took to parse and generate things, can you share those times? I'm using lxml which does much better, that's what we intend to switch breathe to. |
|
Will experiment tomorow with doxygen parameters. Doxygen HTML output works fine, some it may be the XML that is misleading. Timing outputs: Then reading rst sources back takes more than 15 minutes to get around 50% where it crashes Then I get a bazillion of warnings such as |
|
Dear svenevs, I am quite new to Sphinx + Breathe + Exhale but I like your work very much. I was not sure if this is the right "Issue" otherwise, I can generate a new one. I have a similar problem as MrKepzie reported in his last message. My plan is to start documentation for a large existing scientific code (not written by me) which have a lot of non-member functions in *.cpp files. They are declared in *.h and *.hpp files and can thus be used after including the header files everywhere in the code. Unfortunately, this produces the same errors as reported by MrKepzie. For illustration I created a minimal example with the following folder structure: | main.cpp
myclass.cpp
myclass.h
which leads to
I understand that with "#if !defined(DOXYGEN_SHOULD_SKIP_THIS) ..." this issue can be solved in a hardcoded way, but was hoping there is an alternative since it is a quite large code and would hope to find another solution. Therefore I would like to ask if there is an option from Exhale to resolve this issue? I apologize for misleading formulations and in case I used some wrong terms!!! Kind regards |
|
A lot of the problems here are not resolved, and speed for large projects with exhale will need some more serious investigation. That said, a decent number of problems reported here related to signatures of things (functions, and in this case it looks like some macros) could be resolved differently by updating the interface between exhale and breathe. See #106 (comment) for more information. |
|
I generated C++ headers for the Infineon XMC4700 here: https://github.com/vkottler/hal-xmc4700.
It's been running for 83 hours and it's not even 50% done with just the reading/ingesting part... wtf? Source-file count: |
|
I did the same for this project: https://github.com/vkottler/hal-rp2040. https://vkottler.github.io/cpp/sphinx/hal-rp2040/ Took between 1-2 days. I'm happy to wait because the generated output is amazing! Just seems odd that the complexity scaling here is really bad. |
Hi,
We have a large codebase for which we use doxygen. Doxygen generation of XML + HTML is quite fast ~10sec.
When using only breathe + .. doxygenindex :: directive we are able to generate an index (which looks really bad compared to what Doxygen output offers, but that's not the point).
Exhale generates lots of .rst files which then takes more than 30minutes for sphinx to parse in "reading sources...";
This is a deal breaker for us.
I am wondering where the bottleneck is and if something can be done to make it more on par with the performances we find with just doxygenindex directive ?
The text was updated successfully, but these errors were encountered: