Add support for type checking with mypy plugins

[flaeppe]

I'm not totally sure if this approach always works or if there's stuff that I'm missing, but it seems to work. I'll try to explain myself here.

The change utilises pip's --target(docs) argument to point out where to install a project's additional dependencies, passed in via pip_cmd. This replaces the virtual env creation for each project, since that shouldn't be necessary (I think?).

In addition to what's mentioned above we start setting the PYTHONPATH env variable(docs), to point out an additional search path for module files, namely the site-packages directory with the additional dependencies inside the project's virtualenv.

That should allow discovery of project dependencies while keeping the installed mypy versions independent and reusable.

• Closes How to add dependencies with plugins? #8
• Closes do a plugin hack #102

Outdated

---

All this isn't included, but I was thinking that it should be possible to use pip install --target for both mypy installations as well, instead of a virtual env. That leads to having 1 python/pip executable, 1 directory for new_mypy, 1 directory for old_mypy and then all project dependency installations. Although I'm unsure what --python-executable= does and if it could be affected. I'm also not sure how to handle compiled mypy.

Essentially what'll happen with the above changes is that we'll have a directory structure like e.g.

repo-dir/
├── new_mypy
│   └── bin
│       └── mypy
├── old_mypy
│   └── bin
│       └── mypy
├── projects
│   ├── A
│   └── B
└── python
    └── bin
        └── python

We run mypy with something corresponding to

PYTHONPATH="repo-dir/new_mypy:repo-dir/projects/A" repo-dir/new_mypy/bin/mypy --python-executable=repo-dir/python/bin/python ...
PYTHONPATH="repo-dir/old_mypy:repo-dir/projects/A" repo-dir/old_mypy/bin/mypy --python-executable=repo-dir/python/bin/python ...

---

Here's some example debug output:

...
/tmp/mypy_primer/projects/_django-choicefield_venv/bin/pip install . django-stubs pytest         in /tmp/mypy_primer/projects/django-choicefield
PYTHONPATH=/tmp/mypy_primer/new_mypy/venv/lib/python3.12/site-packages:/tmp/mypy_primer/projects/_django-choicefield_venv/lib/python3.12/site-packages
/tmp/mypy_primer/new_mypy/venv/bin/mypy --python-executable=/tmp/mypy_primer/projects/_django-choicefield_venv/bin/python --warn-unused-ignores --warn-redundant-casts --no-incremental --cache-dir=/dev/null --show-traceback --soft-error-limit=-1     in /tmp/mypy_primer/projects/django-choicefield
PYTHONPATH=/tmp/mypy_primer/old_mypy/venv/lib/python3.12/site-packages:/tmp/mypy_primer/projects/_django-choicefield_venv/lib/python3.12/site-packages
/tmp/mypy_primer/old_mypy/venv/bin/mypy --python-executable=/tmp/mypy_primer/projects/_django-choicefield_venv/bin/python --warn-unused-ignores --warn-redundant-casts --no-incremental --cache-dir=/dev/null --show-traceback --soft-error-limit=-1     in /tmp/mypy_primer/projects/django-choicefield
/tmp/mypy_primer/new_mypy/venv/bin/mypy on django-choicefield took 4.96s
/tmp/mypy_primer/old_mypy/venv/bin/mypy on django-choicefield took 4.98s
...

[hauntsaninja]

Thanks for the PR!

I don't quite understand what the new global env is for. Other than that, this PR looks great and does the right thing: we should continue to setup a) a venv for mypy itself, b) project specific venvs. And then use PYTHONPATH (or other) shenanigans so that mypy can import from the project specific venv. We should also add an assert that some version of mypy itself isn't in the project specific venv, otherwise things will get confusing.

[flaeppe]

Thanks for the review, I'm gonna try to move back to using project specific venvs, see if I can get it to work.

The global env was added to have a python/pip binary to reach to when I removed the project specific venvs.

I'll see if I can get a reasonable assert in there too, perhaps just something naive like looking for bin/mypy in a project specific venv is enough.

[flaeppe]

I've updated to align with your suggestions. Let me know if there's anything else I've missed or should update.

Though I can't say I follow why each project needs its own environment. I was thinking it would be preferable to have a virtualenv for each mypy installation and then do a "distributed" install for each project (pip install --target) and via PYTHONPATH= "mount" in the relevant dependencies for each project run. But I might definitely miss something.