[BUG] Incorrect wheel downloaded

[nishantvarma]

Buildout downloads Python 3.8 wheel in Python a 3.10 environment.

We have no distributions for pkg that satisfies 'pkg==7.3.0'.
root: Reading http://admin:[email protected]/pkg/
root: Found link: http://admin:[email protected]/pkg/pkg-7.3.2-py38-none-any.whl
root: Found link: http://admin:[email protected]/pkg/pkg-7.3.0-py310-none-any.whl
root: Found link: http://admin:[email protected]/pkg/pkg-7.3.0-py38-none-any.whl
root: Found link: http://admin:[email protected]/pkg/pkg-7.3.0-py3-none-any.whl
root: pkg==7.3.0 does not match pkg 7.3.2
root: pkg==7.3.0 does not match pkg 7.3.1
Getting distribution for 'pkg==7.3.0'.
Fetching pkg 7.3.0 from: http://admin:[email protected]/pkg/pkg-7.3.0-py38-none-any.whl
root: Downloading http://admin:[email protected]/pkg/pkg-7.3.0-py38-none-any.whl
Got pkg 7.3.0.

Note: We use bytecode-wheels, so version is important.

[nishantvarma]

I also asked in pypa/setuptools#4264. However, it seems to be an issue with how Buildout does things -- it should be able to filter out irrelevant packages much earlier (it does so for C-Python packages etc.). Pip also downloads packages correctly.

[icemac]

This seems to happen since Python 3.8 – for yet unknown reasons.

Please try the following setting, either in ~/.buildout/default.cfg or in your project's buildout.cfg:

[buildout]
abi-tag-eggs = true

It puts the eggs into separate directories depending on their ABI version.

[nishantvarma]

Thanks, @icemac. I tried that -- it still downloads the Python 38 wheel. It does create a folder called eggs/cp310/pkg-7.3.0-py3.10.egg but it the Python 38 contents. I think it should be solved during resolution/download (like pip does for the same package)? Let me know if you have any other suggestions.

As of now, the only workaround I can think of is to pre-fetch the wheels using Pip or something; but that will have to be done outside of Buildout. A simple example is:

for pkg in $(buildout -c buildout.cfg annotate versions | grep "=" |sed "s/= /==/g"); do
    pip download --no-deps --ignore-requires-python -d downloads "$pkg" || true
done

It can even be a plugin.

[gotcha]

Did you check if you would have a cached wrong download ?

This could happen if you have set download-cache in your .buildout/download-cache.

[nishantvarma]

I always set download-cache to downloads/dist. When it is empty, it downloads pkg-py38.whl to cache and converts it as eggs/pkg-py310.eggs. If I manually download pkg-py310.whl to cache, it picks that and converts correctly (it still checks server for something else -- despite disabling newest -- but that doesn't seem to impact) . We wrote a wrapper that populates the cache correctly ATM. I think the download logic has some problems. Didn't get time to check further.