buildout 3 fails on unpinned packages that pip would succeed to install

[odelaere]

Hello,

Testing zc.buildout 3.0.0b3 I noticed a beahavior of pip that should be implemented in zc.buildout.

When you don't pin a specific version of a package, pip tries the latest ant if it fails to install pip would retry the previous version and so on.
ex:

$ bin/pip install "collective.recipe.template"

Collecting collective.recipe.template
  Using cached collective.recipe.template-2.1.tar.gz (13 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   ...
  Downloading collective.recipe.template-2.0.tar.gz (9.3 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   ...
  Downloading collective.recipe.template-1.13.tar.gz (10 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   ...
  Downloading collective.recipe.template-1.12.zip (23 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   ...
  Downloading collective.recipe.template-1.12.tar.gz (11 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   ...
  Downloading collective.recipe.template-1.11.zip (19 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   ...
  Downloading collective.recipe.template-1.10.zip (18 kB)
  Preparing metadata (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   ...
  Downloading collective.recipe.template-1.9.zip (22 kB)
  ...
Successfully installed collective.recipe.template-1.9

Currently zc.buidout does not retry and just fails:

Getting distribution for 'collective.recipe.template==2.1'.
  ERROR: Command errored out with exit status 1:
   command: /srv/zinstances/plonemeeting.portal.core/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-h02pik3h/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-h02pik3h/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-a6ckzh_5
       cwd: /tmp/pip-req-build-h02pik3h/
  Complete output (3 lines):
  /usr/lib/python3.8/distutils/dist.py:274: UserWarning: Unknown distribution option: 'convert_2to3_doctests'
    warnings.warn(msg)
  error in collective.recipe.template setup command: use_2to3 is invalid.
  ----------------------------------------
WARNING: Discarding file:///srv/cache/downloads/dist/collective.recipe.template-2.1.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
No .dist-info directory after successful pip install of /srv/cache/downloads/dist/collective.recipe.template-2.1.tar.gz
While:
  Installing.
  Getting section test-coverage.
  Initializing section test-coverage.
  Installing recipe collective.recipe.template.
  Getting distribution for 'collective.recipe.template==2.1'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/buildout.py", line 2222, in main
    getattr(buildout, command)(args)
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/buildout.py", line 718, in install
    [self[part]['recipe'] for part in install_parts]
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/buildout.py", line 718, in <listcomp>
    [self[part]['recipe'] for part in install_parts]
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/buildout.py", line 1351, in __getitem__
    options._initialize()
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/buildout.py", line 1459, in _initialize
    self.initialize()
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/buildout.py", line 1465, in initialize
    recipe_class = _install_and_load(reqs, 'zc.buildout', entry, buildout)
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/buildout.py", line 1408, in _install_and_load
    zc.buildout.easy_install.install(
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 966, in install
    return installer.install(specs, working_set)
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 691, in install
    for dist in self._get_dist(requirement, ws):
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 582, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 1895, in _move_to_eggs_dir_and_compile
    [tmp_loc] = call_pip_install(dist.location, tmp_dest)
  File "/srv/zinstances/plonemeeting.portal.core/lib/python3.8/site-packages/zc/buildout/easy_install.py", line 1701, in call_pip_install
    distinfo_dir = [
IndexError: list index out of range

[odelaere]

Looks like zc.buildout consider any unpinned package as pinned to the latest version available on pypi

[goschtl]

i guess the setup.py of collective.recipe.template should be fixed.
The use_2to3 is no longer supported in new setuptools versions.

[odelaere]

well that is not the issue

[odelaere]

edit : it's just an example I encountered into this morning

[goschtl]

yes you are right i was to quick
i recently had the same issue with c.r.omelette

[gotcha]

In buildout 3, version selection when unpinned is still done by code from setuptools.

That code should be replaced by code that uses pip version selection.

Hopefully done later