New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Speed up pytype using --precompiled-builtins #1501
Labels
Comments
rchen152
added a commit
to rchen152/typing
that referenced
this issue
Jan 2, 2024
This change gets the pytype test down to ~30s. It's still considerably slower than the others, but any further improvements will likely require untangling the sad state of affairs that is google/pytype#1501, which will take some time. * Runs pytype's 'check' option through pytype.io.check_py to skip the costly type inference that it does by default. * Adds a progress bar so that it doesn't look like pytype is hanging. This generates a few changes to results/pytype/ due to error messages being formatted and parsed in a different way.
JelleZijlstra
pushed a commit
to python/typing
that referenced
this issue
Jan 3, 2024
This change gets the pytype test down to ~30s. It's still considerably slower than the others, but any further improvements will likely require untangling the sad state of affairs that is google/pytype#1501, which will take some time. * Runs pytype's 'check' option through pytype.io.check_py to skip the costly type inference that it does by default. * Adds a progress bar so that it doesn't look like pytype is hanging. This generates a few changes to results/pytype/ due to error messages being formatted and parsed in a different way.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We observed that pytype is surprisingly slow when analyzing https://github.com/google/pycnite, a small project with only 9 .py files. I profiled pytype on a few of the files and discovered that it is spending an unreasonable amount of time (~2s per file) parsing the builtins and typing stubs. We already have a way to parse and pickle these as a preprocessing step (--generate-builtins) and to load the pickle rather than re-parsing every time (--precompiled-builtins).
Perhaps we could generate the builtins in out/bin/pytype/ to speed up the CI workflow and also check them in every time a commit is made to main. I briefly tried to do this and ran into a few issues:
@tests
dirctories in stubs/. This wasn't an issue previously because --generate-builtins was being run with a copy of typeshed into which we'd copied only the files we wanted.The text was updated successfully, but these errors were encountered: