buildout3 rc3 problem

[goschtl]

Hi,

i try to install sqlalchemy via zc.recipe.egg. Buildout3 rc3.

This is my config:

  4 [sqla]
  5 recipe = zc.recipe.egg
  6 eggs = sqlalchemy

I get this error on first run:

While:
  Installing sqla.
  Getting distribution for 'sqlalchemy'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/buildout.py", line 2250, in main
    getattr(buildout, command)(args)
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/buildout.py", line 855, in install
    installed_files = self[part]._call(recipe.install)
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1651, in _call
    return f()
  File "/Users/ck/.buildout/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 227, in install
    reqs, ws = self.working_set()
  File "/Users/ck/.buildout/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 78, in working_set
    ws = self._working_set(
  File "/Users/ck/.buildout/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 161, in _working_set
    ws = zc.buildout.easy_install.install(
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 971, in install
    return installer.install(specs, working_set)
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 696, in install
    for dist in self._get_dist(requirement, ws):
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 581, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 1948, in _move_to_eggs_dir_and_compile
    assert newdist is not None  # newloc above is missing our dist?!
AssertionError

and this on 2'nd

╰ bin/buildout -c tt.cfg
Installing sqla.
Getting distribution for 'sqlalchemy'.
While:
  Installing sqla.
  Getting distribution for 'sqlalchemy'.

An internal error occurred due to a bug in either zc.buildout or in a
recipe being used:
Traceback (most recent call last):
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/buildout.py", line 2250, in main
    getattr(buildout, command)(args)
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/buildout.py", line 855, in install
    installed_files = self[part]._call(recipe.install)
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/buildout.py", line 1651, in _call
    return f()
  File "/Users/ck/.buildout/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 227, in install
    reqs, ws = self.working_set()
  File "/Users/ck/.buildout/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 78, in working_set
    ws = self._working_set(
  File "/Users/ck/.buildout/eggs/zc.recipe.egg-2.0.7-py3.8.egg/zc/recipe/egg/egg.py", line 161, in _working_set
    ws = zc.buildout.easy_install.install(
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 971, in install
    return installer.install(specs, working_set)
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 696, in install
    for dist in self._get_dist(requirement, ws):
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 581, in _get_dist
    dists = [_move_to_eggs_dir_and_compile(dist, self._dest)]
  File "/Users/ck/work/reha/new_bo/lib/python3.9/site-packages/zc/buildout/easy_install.py", line 1922, in _move_to_eggs_dir_and_compile
    os.rename(tmp_loc, newloc)
OSError: [Errno 66] Directory not empty: '/Users/ck/.buildout/eggs/tmp5h568bek/SQLAlchemy-1.4.35-py3.9-macosx-11.5-x86_64.egg' -> '/Users/ck/.buildout/eggs/SQLAlchemy-1.4.35-py3.9-macosx-11.5-x86_64.egg'

any idea?

[mauritsvanrees]

What is suspicious here, is that the tracebacks show a mixture of Python 3.8 (zc.recipe.egg-2.0.7-py3.8.egg) and 3.9.

You tried this with Python 3.9, right?

Can you add abi-tag-eggs = true in your buildout section and try again? I suspect this would solve it. It should work without this option though.

I think I saw a similar report in the issues a while ago, but cannot immediately find it. We could change the default of this option to true. I have this in my ~/.buildout/default.cfg since years.

[gotcha]

Can you try to run it with eggs-directory specified ?

$ buildout buildout:eggs-directory=myeggs install sqla 

In order to avoid old cache ?

(I tried running it on my machine and it did work - even with my eggs cache directory.
However, I must mention there was a time where I had to remove all eggs that had been installed with buildout 2.x.)

[goschtl]

Hey,

first thanks for your help. I guess i found the Problem.
Ok let me try to summarize.

in buildout we have this _get_matching_dist_in_location function. As i see it checks if there is the proper version in the location.

Important: It checks for the platform too.

So buildout grabs on my mac:

SQLAlchemy-1.3.24-py3.9-macosx-11.5-x86_64.egg

but my real platform is

(Pdb++) get_supported_platform()
'macosx-12.2-x86_64'

so there is no match. And it breaks.

What i have done now ist this.

I added the platform=None on the Environment. So the platform check returns True and all works.
I am not sure if it is a bloody hack... Maybe someone has an idea about it.

buildout.py
1869 env = pkg_resources.Environment([location], platform=None)

[gotcha]

I have tried to reproduce your problem (not sure I understood properly).
I did not manage to bump into an issue.
I am not under Mac OSX 12.

This is what I did.

1. make a virtualenv with python 3.9
2. $ bin/pip install zc.buildout==2.13.7
3. run buildout with the following buildout.cfg:

    [sqla]
    recipe = zc.recipe.egg
    eggs = sqlalchemy
    
    [buildout]
    parts = sqla
    abi-tag-eggs = false
    eggs-directory = eggs

Got the following eggs:

drwxr-xr-x   4 gotcha  staff  128 Apr 13 11:35 SQLAlchemy-1.4.35-py3.9-macosx-10.14-x86_64.egg
drwxr-xr-x   5 gotcha  staff  160 Apr 13 11:35 greenlet-1.1.2-py3.9-macosx-10.14-x86_64.egg
drwxr-xr-x   4 gotcha  staff  128 Apr 13 11:35 zc.recipe.egg-2.0.7-py3.9.egg

4. $ bin/pip install zc.buildout==3.0.0rc3
5. $ bin/pip install -U setuptools
6. run buildout again
   Which did use the same eggs:

drwxr-xr-x   4 gotcha  staff  128 Apr 13 11:35 SQLAlchemy-1.4.35-py3.9-macosx-10.14-x86_64.egg
drwxr-xr-x   5 gotcha  staff  160 Apr 13 11:35 greenlet-1.1.2-py3.9-macosx-10.14-x86_64.egg
drwxr-xr-x   4 gotcha  staff  128 Apr 13 11:35 zc.recipe.egg-2.0.7-py3.9.egg

7. remove installed eggs; $ rm -rf eggs
8. run buildout again
   got the same egg versions even if some eggs installed by rc3 are named slightly differently:

drwxr-xr-x   4 gotcha  staff  128 Apr 13 11:38 SQLAlchemy-1.4.35-py3.9-macosx-10.14-x86_64.egg
drwxr-xr-x   5 gotcha  staff  160 Apr 13 11:38 greenlet-1.1.2-py3.9-macosx-10.14-x86_64.egg
drwxr-xr-x   5 gotcha  staff  160 Apr 13 11:38 zc.recipe.egg-2.0.7-py3.9-macosx-10.14-x86_64.egg

@goschtl, what did I miss ?

[goschtl]

Hm,

@gotcha can you gimme a pip list i want to see what version of setuptools and pip you have after you created the virtualenv with python3.9.

Maybe i have Problems with my python setup i use pyenv to have a bunch of python versions on my machine?

Christian

[gotcha]

@goschtl

Before buildout:

Package    Version
---------- -------
pip        22.0.4
setuptools 62.1.0
wheel      0.37.1

With buildout==2.13.7:

Package     Version
----------- -------
pip         22.0.4
setuptools  51.3.3
wheel       0.37.1
zc.buildout 2.13.7

With buildout==3.0.0rc3 and upgraded setuptools:

Package     Version
----------- --------
pip         22.0.4
setuptools  62.1.0
wheel       0.37.1
zc.buildout 3.0.0rc3

[goschtl]

Hm it still does not work here.
The problem is here:

1858     def _get_matching_dist_in_location(dist, location):
1859         """
1860         Check if `locations` contain only the one intended dist.
1861         Return the dist with metadata in the new location.
1862         """
1863         # Getting the dist from the environment causes the distribution
1864         # meta data to be read. Cloning isn't good enough. We must compare
1865         # dist.parsed_version, not dist.version, because one or the other
1866         # may be normalized (e.g., 3.3 becomes 3.3.0 when downloaded from
1867         # PyPI.)
1868         env = pkg_resources.Environment([location])
1869         import pdb; pdb.set_trace()
1870
1871  ->     dists = [ d for project_name in env for d in env[project_name] ]
1872         dist_infos = [ (d.project_name.lower(), d.parsed_version) for d in dists ]
1873         if dist_infos == [(dist.project_name.lower(), dist.parsed_version)]:
1874             return dists.pop()
(Pdb++) [x for x in env]
[]
(Pdb++) new_env_without_platform = pkg_resources.Environment([location], platform=None)
(Pdb++) [x for x in env]
[]
(Pdb++) [x for x in new_env_without_platform]
['sqlalchemy']
(Pdb++) dist
SQLAlchemy 1.4.34 (/var/folders/h1/256d0wp94k529kzp3tfj458w0000gn/T/tmpepliz1ybget_dist/SQLAlchemy-1.4.34-cp39-cp39-macosx_10_15_x86_64.whl)
(Pdb++) location
'/Users/ck/work/sprint/portal/eggs/cp39/SQLAlchemy-1.4.34-py3.9-macosx-11.5-x86_64.egg'
(Pdb++)

You see when i use the Environment with platform = None i get back sqlalchemy. When i use the platform then i get an empty list back. This causes the problem then.

I have no idea what todo...

Christian

[gotcha]

@goschtl Did you try without a shared egg cache, iow, by specifying eggs-directory ?

Any idea why you get 1.3.24 instead of 1.4.35 like me ?

Could you paste the full output of buildout -v ?

Do you still have a mix of python3.9/site-packages/zc/buildout and zc.recipe.egg-2.0.7-py3.8.egg as pointed by @mauritsvanrees, iow the two different python versions ?

You have homework 😉

[goschtl]

Hey i found the thing.

In a brew installed python it works without Problems. But in general i use pyenv to manage my local python installations.

The platform information are a bit different between the pyenv and f.e. brew python installation.

In pyenv i get this:

pkg_resources.get_supported_platform()
'macosx-12.3-x86_64'

in brew python i get this:
macosx-11-x86_64

When i set this platform explicit via this ENV-Variable it works in pyenv python too like a charm:

export _PYTHON_HOST_PLATFORM="macosx-11-x86_64"

I am not sure what it means for buildout itself. But maybe it makes sense to adopt that change here too:

1858 def _get_matching_dist_in_location(dist, location):
1859 """
1860 Check if locations contain only the one intended dist.
1861 Return the dist with metadata in the new location.
1862 """
1863 # Getting the dist from the environment causes the distribution
1864 # meta data to be read. Cloning isn't good enough. We must compare
1865 # dist.parsed_version, not dist.version, because one or the other
1866 # may be normalized (e.g., 3.3 becomes 3.3.0 when downloaded from
1867 # PyPI.)
1868
1869 env = pkg_resources.Environment([location])

1870 dists = [ d for project_name in env for d in env[project_name] ]
1871 dist_infos = [ (d.project_name.lower(), d.parsed_version) for d in dists ]
1872 if dist_infos == [(dist.project_name.lower(), dist.parsed_version)]:
1873 ┆ return dists.pop()

[gotcha]

@goschtl

Two questions:

1. Can we conclude that this is a pyenv bug ?
2. Can you explain what this means ?

maybe it makes sense to adopt that change here too

[idgserpro]

I do believe that this is similar to pypa/setuptools#2514 (comment): a macos + setuptools problem.

[davisagli]

I ran into this error today. I'm not sure I have much to add yet on understanding the cause. But, I also have Python installed using pyenv. I tried reinstalling Python with pyenv (well, actually, I made a new installation of Python 3.9.16, which is slightly newer than the Python 3.9.13 I had before). With the new Python, the error went away.

So, I am guessing this has something to do with a mismatch between the version of MacOS that was used to build Python with pyenv, and the version that is currently installed (i.e. after upgrading MacOS you need to rebuild Python)

[mauritsvanrees]

@davisagli wrote:

I ran into this error today. I'm not sure I have much to add yet on understanding the cause. But, I also have Python installed using pyenv. I tried reinstalling Python with pyenv (well, actually, I made a new installation of Python 3.9.16, which is slightly newer than the Python 3.9.13 I had before). With the new Python, the error went away.

So, I am guessing this has something to do with a mismatch between the version of MacOS that was used to build Python with pyenv, and the version that is currently installed (i.e. after upgrading MacOS you need to rebuild Python)

I think you are right. I upgraded from MacOS 12 to 13.5. Rerunning a Plone buildout failed on msgpack. I could reproduce it with a very simple buildout.cfg:

[buildout]
extends = https://dist.plone.org/release/6.0-dev/versions.cfg
parts = test

[test]
recipe = zc.recipe.egg
eggs = msgpack

It failed at assert newdist is not None, same as in a traceback above. The egg location was msgpack-1.0.5-py3.11-macosx-12.6-x86_64.egg.
Then I reinstalled my Python (pyenv install -f 3.11.4) and tried again. Now an egg got installed as msgpack-1.0.5-py3.11-macosx-13.5-x86_64.egg and it works fine.

Now in my Plone 6.0 core development buildout after I rerun buildout, I have 25 eggs with macosx-13.5 in their names. In my not yet updated Plone 6.1 buildout I still see 25 macosx-12.6 eggs. Updated it: now 6.1 also has only macosx-13.5 eggs`.

[mauritsvanrees]

TLDR: if building a wheel fails, you may need to clear your buildout download cache.

I now have a problem in a buildout that tries to install zope.container 4.10 on Python 3.8.
See also this answer.

Running buildout with high verbosity, after a while I get this:

root: Reading https://pypi.org/simple/zope.container/
root: Found link: https://files.pythonhosted.org/... lots of links.
...
root: zope.container==4.10 does not match zope.container 5.1
root: zope.container==4.10 does not match zope.container 5.1
root: zope.container==4.10 does not match zope.container 5.1
root: zope.container==4.10 does not match zope.container 5.0
root: zope.container==4.10 does not match zope.container 5.0
Getting distribution for 'zope.container==4.10'.
Download cache has zope.container 4.10 at: /Users/maurits/cached-downloads/dist/zope.container-4.10.tar.gz
Running pip install:
"/Users/maurits/clients/zeelandia/plone52/bin/python3.8" "-m" "pip" "install" "--no-deps" "-t" "/Users/maurits/shared-eggs/cp38/tmpc4r2gd3k" "-v" "/Users/maurits/cached-downloads/dist/zope.container-4.10.tar.gz"
...
  running build_ext
  building 'zope.container._zope_container_contained' extension
  creating build/temp.macosx-13.5-x86_64-cpython-38
  creating build/temp.macosx-13.5-x86_64-cpython-38/src
  creating build/temp.macosx-13.5-x86_64-cpython-38/src/zope
  creating build/temp.macosx-13.5-x86_64-cpython-38/src/zope/container
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -DOPENSSL_NO_SSL3 -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/include -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/include -I/usr/local/opt/openssl/include -I/usr/local/opt/readline/include -I/usr/local/include -I/usr/local/opt/zlib/include -I/Users/maurits/.pyenv/versions/3.8.17/include/python3.8 -I/Users/maurits/.pyenv/versions/3.8.17/include/python3.8 -I/private/var/folders/26/1plvhxbs6yx7g_82v2xdxc500000gn/T/pip-req-build-41be7qzg/.eggs/zope.proxy-5.0.0-py3.8-macosx-13.5-x86_64.egg/zope/proxy -I/private/var/folders/26/1plvhxbs6yx7g_82v2xdxc500000gn/T/pip-req-build-41be7qzg/.eggs/persistent-5.0-py3.8-macosx-13.5-x86_64.egg/persistent -I/Users/maurits/clients/zeelandia/plone52/include -I/Users/maurits/.pyenv/versions/3.8.17/include/python3.8 -c src/zope/container/_zope_container_contained.c -o build/temp.macosx-13.5-x86_64-cpython-38/src/zope/container/_zope_container_contained.o
...
  src/zope/container/_zope_container_contained.c:115:11:
 error: implicit declaration of function 'PyString_AS_STRING' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
...
  4 warnings and 4 errors generated.
  error: command '/usr/bin/clang' failed with exit code 1
  error: subprocess-exited-with-error
  
  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
...
Failed to build zope.container
ERROR: Could not build wheels for zope.container, which is required to install pyproject.toml-based projects

That last remark might be a wrong conclusion in either buildout or pip: zope.container is not pyproject.toml-based.

But why does it try to build from source, instead of using a pre-built wheel that it can download from PyPI? When you look at the pypi simple page and do some prepping, you get these Mac OSX specific wheels:

$ grep cp311 zope-container.html | grep macosx | cut -d '"' -f2 | cut -d "/" -f8- | cut -d "#" -f1
zope.container-4.8-cp311-cp311-macosx_11_0_x86_64.whl
zope.container-4.10-cp311-cp311-macosx_10_9_x86_64.whl
zope.container-4.10-cp311-cp311-macosx_11_0_arm64.whl
zope.container-5.0-cp311-cp311-macosx_10_9_x86_64.whl
zope.container-5.0-cp311-cp311-macosx_11_0_arm64.whl
zope.container-5.1-cp311-cp311-macosx_10_9_x86_64.whl
zope.container-5.1-cp311-cp311-macosx_11_0_arm64.whl

In my case I need the x86_64 wheels. And then the strange thing is:

• buildout of zope.container 5.1 works fine.
• pip install of 4.10 works fine.

So the wheel should be compatible, and buildout should recognise this, but buildout of 4.10 fails. I could use one of the two working solutions as workaround, but still: why does buildout 4.10 fail?

Okay, I try with a very simple buildout, and do not use the shared buildout caches:

[buildout]
extends = https://zopefoundation.github.io/Zope/releases/4.8.7/versions.cfg
parts = test
eggs-directory = eggs
download-cache = downloads
abi-tag-eggs = true

[test]
recipe = zc.recipe.egg
interpreter = zopepy
eggs = zope.container

Hey, it works!

What is the influence of the caches? My shared cache only contains zope.container-4.10.tar.gz and no wheel. The cache in this fresh buildout does contain the wheel.

When I remove the egg, and copy the tarball to the download cache, it still works: the wheel is in the download cache. Remove both the egg and the wheel, keep only the tarball: it breaks.

Of course in principle it should be possible to build from source as well, but this does not always work, and I may need to install more development libraries or set more environment variables.

So after upgrading Mac to a new version:

1. Reinstall all Pythons with pyenv.
2. Clear/rename the buildout download cache.
3. Probably clear/rename the buildout eggs cache as well.

Still, probably something could be improved in buildout. If it has a cached source tarball, it should still check PyPI for an available wheel, and prefer that one.

[gotcha]

Thanks for the detailed analysis !

[fredvd]

This issue got on my radar as well since the release of Mac OS X Sonoma, where I have been battling with a Plone 5.2 buildout setup with zc.buildout 3 that worked perfectly fine before, but started throwing both exceptions in October/November. (one or the other, depending on what I tried to fix with clearing/isolating caches etc). That's:

assert newdist is not None # newloc above is missing our dist?! - AssertionError and: OSError: [Errno 66] Directory not empty:

This first happened on an M1 mac. And when I needed to travel and I wanted to update on my Intel Macbook it happened again. That really surprised me as in the last 3 years I've only had issues on the new apple Arm architecture.

The solution that worked for me right away was doing an
export _PYTHON_HOST_PLATFORM="macosx-14-x86_64" on my intel laptop with OSX 14 Sonoma and all bin/buildout
issues disappeared.

This is a very nasty issue to debug and I was initially surprised it took me this long to figure this out (and also find these reports) with all the experience and issues I have seen in the last 3 years when I switched to Arm based Macs. I think because the errors are so vague and generic: I asked @mauritsvanrees about these tracebacks and allthough he worked on it, he even didn't remember that he saw these tracebacks not longer than 3 months ago.....

---

Sorry: forgot a part, discussed this with Maurits this week: the other 'solution' or missing step here is that pyenv built pythons on a previous OS X version have the wrong PLATFORM stored in the installation. So either you force the platform by overriding in the environment variable, or you rebuild the pyenv Python and when installing packages it will use the correct current platform it was built with.

Then apparently a 12.X built python can and should use 14.X created wheels/eggs and that's not a problem? Or am I playing with fire by overriding it with the environment variable?