pip install w/ extras does not compute version constraints correctly

[tych0]

We recently started seeing CI failures, e.g. https://github.com/qtile/qtile/actions/runs/9141328074/job/25135602998?pr=4817

Full error for posterity:

2024-05-18T16:55:17.6039189Z pypy3-x11: install_deps> python -I -m pip install 'cairocffi>=1.6.0' cffi coverage dbus-next isort 'libcst>=1.0.0' PyGObject 'pytest>=6.2.1' 'setuptools>=40.5.0' wheel 'xcffib>=1.4.0'
2024-05-18T16:56:41.1700758Z pypy3-x11: freeze> python -m pip freeze --all
2024-05-18T16:56:41.9691619Z pypy3-x11: cairocffi==1.7.0,cffi==1.17.0.dev0,coverage==7.5.1,dbus-next==0.2.3,exceptiongroup==1.2.1,greenlet==0.4.13,hpy==0.9.0,iniconfig==2.0.0,isort==5.13.2,libcst==1.3.1,packaging==24.0,pip==24.0,pluggy==1.5.0,pycairo==1.26.0,PyGObject==3.48.2,pytest==8.2.0,PyYAML==6.0.1,readline==6.2.4.1,setuptools==69.5.1,tomli==2.0.1,wheel==0.43.0,xcffib==1.5.0
2024-05-18T16:56:41.9717925Z ##[group]tox: pypy3-x11
2024-05-18T16:56:41.9718750Z pypy3-x11: commands[0]> pip install .
2024-05-18T16:56:42.8863402Z Processing /home/runner/work/qtile/qtile
2024-05-18T16:56:42.8938838Z   Installing build dependencies: started
2024-05-18T16:56:46.1516903Z   Installing build dependencies: finished with status 'done'
2024-05-18T16:56:46.1536479Z   Getting requirements to build wheel: started
2024-05-18T16:56:46.4918314Z   Getting requirements to build wheel: finished with status 'done'
2024-05-18T16:56:46.5032122Z   Installing backend dependencies: started
2024-05-18T16:56:54.4156218Z   Installing backend dependencies: finished with status 'error'
2024-05-18T16:56:54.4394563Z   error: subprocess-exited-with-error
2024-05-18T16:56:54.4395177Z   
2024-05-18T16:56:54.4398271Z   × pip subprocess to install backend dependencies did not run successfully.
2024-05-18T16:56:54.4399553Z   │ exit code: 1
2024-05-18T16:56:54.4400088Z   ╰─> [36 lines of output]
2024-05-18T16:56:54.4400591Z       Collecting cffi>=1.1.0
2024-05-18T16:56:54.4401083Z         Downloading cffi-1.16.0.tar.gz (512 kB)
2024-05-18T16:56:54.4401924Z            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 512.9/512.9 kB 9.0 MB/s eta 0:00:00
2024-05-18T16:56:54.4402934Z         Installing build dependencies: started
2024-05-18T16:56:54.4403719Z         Installing build dependencies: finished with status 'done'
2024-05-18T16:56:54.4404358Z         Getting requirements to build wheel: started
2024-05-18T16:56:54.4405254Z         Getting requirements to build wheel: finished with status 'done'
2024-05-18T16:56:54.4406272Z         Installing backend dependencies: started
2024-05-18T16:56:54.4407485Z         Installing backend dependencies: finished with status 'done'
2024-05-18T16:56:54.4408403Z         Preparing metadata (pyproject.toml): started
2024-05-18T16:56:54.4409584Z         Preparing metadata (pyproject.toml): finished with status 'done'
2024-05-18T16:56:54.4410595Z       Collecting cairocffi>=1.6.0 (from cairocffi[xcb]>=1.6.0)
2024-05-18T16:56:54.4411435Z         Using cached cairocffi-1.7.0-py3-none-any.whl.metadata (3.3 kB)
2024-05-18T16:56:54.4412076Z       Collecting xcffib>=1.4.0 (from cairocffi[xcb]>=1.6.0)
2024-05-18T16:56:54.4412780Z         Using cached xcffib-1.5.0-py3-none-any.whl
2024-05-18T16:56:54.4413818Z       INFO: pip is looking at multiple versions of xcffib to determine which version is compatible with other requirements. This could take a while.
2024-05-18T16:56:54.4414794Z         Downloading xcffib-1.4.0.tar.gz (89 kB)
2024-05-18T16:56:54.4415993Z            ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 89.5/89.5 kB 15.2 MB/s eta 0:00:00
2024-05-18T16:56:54.4416780Z         Preparing metadata (setup.py): started
2024-05-18T16:56:54.4417383Z         Preparing metadata (setup.py): finished with status 'done'
2024-05-18T16:56:54.4418000Z       Collecting cairocffi[xcb]>=1.6.0
2024-05-18T16:56:54.4418602Z         Downloading cairocffi-1.6.1-py3-none-any.whl.metadata (3.3 kB)
2024-05-18T16:56:54.4419334Z         Downloading cairocffi-1.6.0-py3-none-any.whl.metadata (3.3 kB)
2024-05-18T16:56:54.4420373Z       INFO: pip is looking at multiple versions of cairocffi[xcb] to determine which version is compatible with other requirements. This could take a while.
2024-05-18T16:56:54.4421704Z       INFO: pip is still looking at multiple versions of xcffib to determine which version is compatible with other requirements. This could take a while.
2024-05-18T16:56:54.4422960Z       ERROR: Cannot install cairocffi>=1.6.0 and cairocffi[xcb]==1.7.0 because these package versions have conflicting dependencies.
2024-05-18T16:56:54.4423737Z       
2024-05-18T16:56:54.4424059Z       The conflict is caused by:
2024-05-18T16:56:54.4424501Z           The user requested cairocffi>=1.6.0
2024-05-18T16:56:54.4426690Z           cairocffi[xcb] 1.7.0 depends on cairocffi 1.7.0 (from https://files.pythonhosted.org/packages/4c/0d/c9b8a12971276b33143422ff6b73aa7d74cff5f7effbffbe2a1df3bd6590/cairocffi-1.7.0-py3-none-any.whl (from https://pypi.org/simple/cairocffi/) (requires-python:>=3.8))
2024-05-18T16:56:54.4428113Z       
2024-05-18T16:56:54.4428435Z       To fix this you could try to:
2024-05-18T16:56:54.4429077Z       1. loosen the range of package versions you've specified
2024-05-18T16:56:54.4429752Z       2. remove package versions to allow pip attempt to solve the dependency conflict
2024-05-18T16:56:54.4430367Z       
2024-05-18T16:56:54.4431327Z       ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
2024-05-18T16:56:54.4432174Z       [end of output]
2024-05-18T16:56:54.4432496Z   
2024-05-18T16:56:54.4433079Z   note: This error originates from a subprocess, and is likely not a problem with pip.
2024-05-18T16:56:54.4467791Z error: subprocess-exited-with-error
2024-05-18T16:56:54.4468321Z 
2024-05-18T16:56:54.4468977Z × pip subprocess to install backend dependencies did not run successfully.
2024-05-18T16:56:54.4470301Z │ exit code: 1
2024-05-18T16:56:54.4471009Z ╰─> See above for output.
2024-05-18T16:56:54.4471408Z 
2024-05-18T16:56:54.4471989Z note: This error originates from a subprocess, and is likely not a problem with pip.
2024-05-18T16:56:54.4811842Z pypy3-x11: exit 1 (12.51 seconds) /home/runner/work/qtile/qtile> pip install . pid=5509
2024-05-18T16:56:54.4825581Z ##[endgroup]
2024-05-18T16:56:54.4826776Z   pypy3-x11: FAIL code 1 (97.43=setup[84.93]+cmd[12.51] seconds)
2024-05-18T16:56:54.4827684Z   evaluation failed :( (98.05 seconds)
2024-05-18T16:56:54.4928083Z ##[error]Process completed with exit code 1.

It looks to me like the problem is that the solver can't "see" cairocffi[xcb] == 1.7.0 satisfies cairocffi >= 1.6.0. It is not clear why this just started happening to us now, given that cairocffi 1.7.0 was released three weeks ago. We also only see this behavior on pypy, which is why I filed the bug here vs. pip directly. Let me know if I should move it there.

[cfbolz]

yeah, tbh it sounds more like a question for pip, very unlikely that pypy is to blame here. feel free to leave the issue open, but maybe also open a pip one?

[mattip]

Can you cut this down to a smaller reproducer? When I try this piece-by-piece on pypy3.9-v7.3.16 on linux64, I get different errors, even after updating pip to 24.0

$ pypy -m pip install PyGObject
...
Collecting PyGObject
  Downloading pygobject-3.48.2.tar.gz (715 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 715.2/715.2 kB 9.7 MB/s eta 0:00:00
...
Run-time dependency gobject-introspection-1.0 found: NO (tried pkgconfig and cmake)
      Not looking for a fallback subproject for the dependency gobject-introspection-1.0 because:
      Use of fallback dependencies is disabled.
      
      ../meson.build:29:9: ERROR: Dependency 'gobject-introspection-1.0' is required but not found.

$ python -m pip install 'xcffib>=1.4.0'
Collecting xcffib>=1.4.0
  Using cached xcffib-1.5.0.tar.gz (89 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Installing backend dependencies ... error
  error: subprocess-exited-with-error
  
  × pip subprocess to install backend dependencies did not run successfully.
  │ exit code: 1
  ╰─> [5 lines of output]
      Collecting wheel
        Using cached wheel-0.43.0-py3-none-any.whl.metadata (2.2 kB)
      ERROR: Ignored the following yanked versions: 1.0.2.post2
      ERROR: Could not find a version that satisfies the requirement cffi==1.17.0.dev0 (from versions: 0.1, 0.2, 0.2.1, 0.3, 0.4, 0.4.1, 0.4.2, 0.5, 0.6, 0.7, 0.7.1, 0.7.2, 0.8, 0.8.1, 0.8.2, 0.8.3, 0.8.4, 0.8.5, 0.8.6, 0.9.0, 0.9.1, 0.9.2, 1.0.0, 1.0.1, 1.0.3, 1.1.0, 1.1.1, 1.1.2, 1.2.0.post1, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1, 1.4.2, 1.5.0, 1.5.1, 1.5.2, 1.6.0, 1.7.0, 1.8.2, 1.8.3, 1.9.0, 1.9.1, 1.10.0, 1.11.0, 1.11.1, 1.11.2, 1.11.3, 1.11.4, 1.11.5, 1.12.0, 1.12.1, 1.12.2, 1.12.3, 1.13.0, 1.13.1, 1.13.2, 1.14.0, 1.14.1, 1.14.2, 1.14.3, 1.14.4, 1.14.5, 1.14.6, 1.15.0rc1, 1.15.0rc2, 1.15.0, 1.15.1, 1.16.0rc1, 1.16.0rc2, 1.16.0, 1.17.0rc1)
      ERROR: No matching distribution found for cffi==1.17.0.dev0

This seems to come from a recent release of cffi==1.17.0rc1 on May 24.

Installing cairoffi does succeed:

$ python -m pip install 'cairocffi>=1.6.0'
Collecting cairocffi>=1.6.0
  Using cached cairocffi-1.7.0-py3-none-any.whl.metadata (3.3 kB)
Requirement already satisfied: cffi>=1.1.0 in /home/matti/oss/pypy3.9-v7.3.16-linux64/lib/pypy3.9 (from cairocffi>=1.6.0) (1.17.0.dev0)
Downloading cairocffi-1.7.0-py3-none-any.whl (75 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 75.4/75.4 kB 793.2 kB/s eta 0:00:00
Installing collected packages: cairocffi
Successfully installed cairocffi-1.7.0

[mattip]

The xcffifb issue is tych0/xcffib#166 which was fixed in tych0/xcffib#167.

[tych0]

I see, so I think this is close-able then. (I wonder if pip should special-case ignore cffi on pypy so others don't stumble over this?)