ModuleNotFoundError: No module named 'setuptools.command.test'

[Svalorzen]

I have a PyBind-based project which I build using scikit-build-core in order to use the resulting wrappers from Python. Today, building the wheel for my project fails with the error

ModuleNotFoundError: No module named 'setuptools.command.test'

As far as I've been able to understand, this is due to a breaking change in setuptools version 72; here is the bug report for it.

I'm not sure whether I could configure my project in a way that would bypass this issue, and more generally, that would pin dependencies in such a way that I don't risk having such problems again. I have tried having my venv install setuptools==71.1.0 but the issue persisted (something that is mentioned in the linked issue as well).

In case it helps, here is my pyproject.toml:

[build-system]
requires = ["scikit-build-core>=0.3.3", "pybind11"]
build-backend = "scikit_build_core.build"

[project]
name = "####"
version = "0.0.1"
description="#######"
readme = "README.md"
requires-python = ">=3.10"
dependencies = ["numpy"]

[tool.scikit-build]
wheel.expand-macos-universal-tags = true
cmake.build-type = "Debug"

[LecrisUT]

Thanks for the report. Nothing we can do on our end since it's an issue with any dependency that uses setuptools as the backend. Need to wait at least for setuptools 72.0.0 to be yanked

[Svalorzen]

I see, but perhaps there might be (or maybe there ought to be) a way to fix the setuptools version, so one can pin the build environment?

[LecrisUT]

This would depend on how pip deals with different build-system.requires constraints in build-issolation. Maybe having a dummy project in there that pins setuptools!=72.0.0 could help propagate that. Otherwise we'll have to be patient. I have pinged the discord if there is anyone present to yank that.

[LecrisUT]

@Svalorzen Actually, can you share your build log on how you got that error? The other case I've encountered was on cssbeautfier where it genuinely used the broken api ¹, but for pybind11, I don't see it being used, and cmake and ninja use scikit-build-core which shouldn't be affected by this.

Footnotes

1. https://github.com/beautifier/js-beautify/blob/0863e19a3cd2bcbaf2483ec1836c015a643ba68d/python/setup-css.py#L9 ↩

[Svalorzen]

  Resolved https://github.com/my_account/my_project to commit yyyyyyyyyyyyyyyyyyyyyyy
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Preparing metadata (pyproject.toml): started
  Preparing metadata (pyproject.toml): finished with status 'done'
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'error'
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [17 lines of output]
      Traceback (most recent call last):
        File "/home/runner/work/xxxxxxxxxx/xxxxxx/build/_deps/zzzzzz-src/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/runner/work/xxxxxxxxxx/xxxxxx/build/_deps/zzzzzz-src/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/runner/work/xxxxxxxxxx/xxxxxx/build/_deps/zzzzzz-src/venv/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 118, in get_requires_for_build_wheel
          return hook(config_settings)
        File "/tmp/pip-build-env-fnom811n/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 327, in get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=[])
        File "/tmp/pip-build-env-fnom811n/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 297, in _get_build_requires
          self.run_setup()
        File "/tmp/pip-build-env-fnom811n/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 497, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/tmp/pip-build-env-fnom811n/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 313, in run_setup
          exec(code, locals())
        File "<string>", line 9, in <module>
      ModuleNotFoundError: No module named 'setuptools.command.test'
      [end of output]

Hope this helps! The first few messages might be intertwined with the successful installation of other packages; I'm not sure.

I admit that I'm not a huge expert on how to config Python projects; if this is something I can configure in the project that tries to install my scikit dependency do let me know; as I said, in my CI I tried directly installing only setuptools 71 but the error did not change..

[LecrisUT]

I think you've obstructed the crucial part that would tell us which dependency is affected. For example in the following:

 An unexpected error has occurred: CalledProcessError: command: ('/home/runner/.cache/pre-commit/repof46ludlk/py_env-python3.12/bin/python', '-mpip', 'install', '.')
return code: 1
stdout:
    Processing /home/runner/.cache/pre-commit/repof46ludlk
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'
      Preparing metadata (pyproject.toml): started
      Preparing metadata (pyproject.toml): finished with status 'done'
    Collecting PyYAML<7.0,>=6.0 (from djlint==1.34.1)
      Using cached PyYAML-6.0.1-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
    Collecting click<9.0.0,>=8.0.1 (from djlint==1.34.1)
      Using cached click-8.1.7-py3-none-any.whl.metadata (3.0 kB)
    Collecting colorama<0.5.0,>=0.4.4 (from djlint==1.34.1)
      Downloading colorama-0.4.6-py2.py3-none-any.whl.metadata (17 kB)
    Collecting cssbeautifier<2.0.0,>=1.14.4 (from djlint==1.34.1)
      Downloading cssbeautifier-1.15.1.tar.gz (25 kB)
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'error'
stderr:
      error: subprocess-exited-with-error
      
      × Getting requirements to build wheel did not run successfully.
      │ exit code: 1

I've found the culprit to be cssbeautifier. The important part were the Collecting and Downloading parts there.

[Svalorzen]

Ah I see what you mean. The error might be triggered by any dependency, and not simply the scikit-build-core one. I was assuming that because the error mentioned build wheel that it must have been my own project that broke (since it needs to build/compile the C++ code).

How did you figure out that it was cssbeautifier? What gives away which dependency/ies was/were triggering the error?

[LecrisUT]

Basically each dependency was being handled in order, and I've noticed that it was trying to resolve cssbeautifier due to djlint (the main package being installed). The clue was that this issue should not happen if you are using a "pre-built" wheel, and I found it weird that djlint would be erroring out on that. Then I realized it was a red-herring and when I looked more closely at the logs and I figured what was actually going on.

[henryiii]

It cannot be triggered by scikit-build-core, because we don't use setuptools.

Ninja still uses scikit-build classic, there hasn't been a release since we moved it to scikit-build-core. Also pybind11 uses setuptools still, but that's a pure wheel, and it doesn't use the test command. Oh, your only dep is numpy? That does't use setuptools either. Very strange!

This is happening on line 9 of a setup.py somewhere, so I'm guessing it's not either of those, though, as you'd see skbuild in the error (and pybind11 don't use this command in it's setup.py). I bet it's some other dependency, maybe an install dependency? You could try building it with PyPA's build to see if it's happening in the install phase, which I'd expect.

Though setuptools 72 has been yanked so unless you set a PIP_CONSTRAINTS to force it, it will just work for now. FYI, in the future, setting PIP_CONSTRAINTS does affect the isolated environment, so it can be used for workarounds.