How to add dependencies with plugins?

[sobolevn]

Hi! Thanks a lot for this tool!

I am trying to add https://github.com/dry-python/returns support (because we have quite a lot of complex plugins).

Code:

    Project(
        location="https://github.com/dry-python/returns.git",
        mypy_cmd="{mypy} returns",
        pip_cmd="{pip} install .",
        expected_success=True,
    ),

returns is based on poetry, not pip. But pip is able to install everything from pyproject.toml locally.

The next problem is that our plugin is not found:

old mypy
> /tmp/mypy_primer/old_mypy/venv/bin/mypy returns --python-executable=/tmp/mypy_primer/projects/_returns_venv/bin/python --no-incremental --cache-dir=/dev/null
UNEXPECTED FAILURE
        setup.cfg:153: error: Error importing plugin 'returns.contrib.mypy.returns_plugin': No module named 'returns'
        Found 1 error in 1 file (errors prevented further checking)
----------

Looks like the problem comes from a fact that mypy and returns use different venvs. But, I am not sure how to fix this.

[hauntsaninja]

Yeah, mypy_primer doesn't yet support plugins — you'll notice mention of dry-python/returns at

mypy_primer/mypy_primer.py

Line 1249 in 7c00c13

# Other repos with plugins: dry-python/returns, strawberry-graphql/strawberry, edgedb/edgedb

:-) — and I'm not sure what the best way to support them is either.

The difficulty, as you mentioned, is that mypy and the projects are installed in different venvs. This allows us to reuse the same mypy venv across projects / easily change the mypy version independently.

I think one (possibly overkill) way of doing this is if we can get the project specific venv to "inherit" the mypy installation from the mypy venv. I bet there's some fun .pth hack way to do this.

[sobolevn]

you'll notice mention of dry-python/returns at

@hauntsaninja yeah, this was a pleasant surprise! 🙂

I think one (possibly overkill) way of doing this is if we can get the project specific venv to "inherit" the mypy installation from the mypy venv. I bet there's some fun .pth hack way to do this.

I think that it is also possible to install mypy on top of the returns (for example) venv. Like /tmp/returns_env/bin/pip install /tmp/mypy_venv/path/to/mypy/setup.py
And then run it /tmp/returns_env instead of /tmp/mypy_env

[sobolevn]

Yes, it works after a quick experiment:

» /tmp/mypy_primer/projects/_returns_venv/bin/pip install /tmp/mypy_primer/old_mypy/mypy_master
...
Successfully installed mypy-0.820+dev.a503132749eb179eec8d1e97f6b8e372daab6ce7

» /tmp/mypy_primer/projects/_returns_venv/bin/mypy returns/returns --ignore-missing-imports     
Success: no issues found in 112 source files

[hauntsaninja]

Yeah, that's possible, it's just a little tricky to get right. E.g., you'd need to prevent "old mypy" and "new mypy" from running in parallel on the same project. There are some other gotchas too (bisection, using mypy pypi wheels).

[sobolevn]

Maybe we can add something like

[mypy_primer]
build_command = 'make && make install'
# possibly `remove_command` as well

to plugins' mypy.ini / setup.cfg / pyproject.toml?

This will just contain the command we need to install this in mypy_primer's env.
So, this will be an opt-in feature for plugin authors.

For example, I am very worried about my projects are not tested (because the most complex ones break the most). And I would totally opt-in.

[pranavrajpal]

Would specifying the plugin by path instead of module name work like what edgedb does right now? edgedb/edgedb#2627 seems to suggest that doing that makes installing the plugin unnecessary.

[flaeppe]

I just wanted to mention on this issue that I think I did figure out an approach to make mypy_primer support plugin dependencies in #121