-
Notifications
You must be signed in to change notification settings - Fork 113
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Radon struggles to collect statistics from large files #223
Comments
Hi Sam, thanks for sharing the example. Indeed, it's quite surprising to see such a long run time for such a simple file. The However, the superlinear complexity is definitely in Radon's code. It performs some complicated operations to count logical lines, and I suspect that's where the slowest code is. I think your example highlights one of the inefficiencies particularly well. The next step would be to profile the code. A flamegraph should already give some very useful hints. I'll try to investigate this when I've got time. |
Thanks for the info, that's really helpful! |
I have some large files that radon struggles to analyse. I created an example to demonstrate the problem: https://gist.githubusercontent.com/Sam152/50e8ef27cceb899084b42a069237a7b8/raw/bb21870395df86a0062c22353b532b45d31bd3f5/sample.py (~800 lines)
In my case running
radon raw big-package
takes 28.38s. In reality the module I'm trying to analyse has ~ 5000 lines with a similar amount of AST per line.If I double my 800 line example, the script takes roughly 115.50s to run, so my feeling is that there might be something which scales worse than
O(n)
per-AST.Any pointers if there might be something that can be optimised here, or if the nature of the analysis is such that speeding this process up is simply not possible?
Thanks in advance, if anyone can share their experience.
Cheers,
Sam
On a side note, while researching this issue, I found radon cited in an academic paper, which I thought was interesting and worth sharing (https://arxiv.org/pdf/2007.08978.pdf).
The text was updated successfully, but these errors were encountered: