Running cythonize failed!

[poopkinel]

• Contextualized Topic Models version: current to 25/11/2023
• Python version: 3.12
• Operating System: Windows

Description

I wanted to pip install the module.
I downloaded Python3.12 to a new computer (windows). I installed pip with get-pip.py from the Python website. I ran pip install -U contextualized_topic_models. The installation started and failed at:

Preparing metadata (pyproject.toml) did not run successfully.
....
RuntimeError: Running cythonize failed!

I expected the installation to finish successfully, installing all the dependencies.

I tried installing numpy myself: pip install numpy
then I installed Cython myself: pip install Cython

both were successful.
Then I tried installing CTM again with
pip install -U contextualized_topic_models
and I got the same response:
RuntimeError: Running cythonize failed!

`pip install -U contextualized_topic_models`

Collecting contextualized_topic_models
  Using cached contextualized_topic_models-2.5.0-py2.py3-none-any.whl (36 kB)
Requirement already satisfied: numpy>=1.19.1 in c:\users\ophek\appdata\local\programs\python\python312\lib\site-packages (from contextualized_topic_models) (1.26.2)
INFO: pip is looking at multiple versions of contextualized-topic-models to determine which version is compatible with other requirements. This could take a while.
  Using cached contextualized_topic_models-2.4.2-py2.py3-none-any.whl (35 kB)
  Using cached contextualized_topic_models-2.4.1-py2.py3-none-any.whl (35 kB)
  Using cached contextualized_topic_models-2.4.0-py2.py3-none-any.whl (35 kB)
  Using cached contextualized_topic_models-2.3.0-py2.py3-none-any.whl (35 kB)
  Using cached contextualized_topic_models-2.2.1-py2.py3-none-any.whl (34 kB)
  Using cached contextualized_topic_models-2.2.0-py2.py3-none-any.whl (33 kB)
  Using cached contextualized_topic_models-2.1.2-py2.py3-none-any.whl (33 kB)
INFO: pip is still looking at multiple versions of contextualized-topic-models to determine which version is compatible with other requirements. This could take a while.
  Using cached contextualized_topic_models-2.1.1-py2.py3-none-any.whl (33 kB)
  Using cached contextualized_topic_models-2.0.1-py2.py3-none-any.whl (31 kB)
  Using cached contextualized_topic_models-2.0.0-py2.py3-none-any.whl (31 kB)
  Using cached contextualized_topic_models-1.8.2-py2.py3-none-any.whl (29 kB)
Collecting numpy==1.19.1 (from contextualized_topic_models)
  Using cached numpy-1.19.1.zip (7.3 MB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error

  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [92 lines of output]
      Running from numpy source directory.
      <string>:470: UserWarning: Unrecognized setuptools command, proceeding with generating Cython sources and expanding templates
      C:\Users\Ophek\AppData\Local\Temp\pip-install-n406nd2p\numpy_6a1cb8d904c14ee5920e5b7a6fef2fa6\tools\cythonize.py:73: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
        required_version = LooseVersion('0.29.21')
      C:\Users\Ophek\AppData\Local\Temp\pip-install-n406nd2p\numpy_6a1cb8d904c14ee5920e5b7a6fef2fa6\tools\cythonize.py:75: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
        if LooseVersion(cython_version) < required_version:
      performance hint: _common.pyx:275:19: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:299:19: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:322:50: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:426:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:465:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:509:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:592:36: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:596:36: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:600:36: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:604:36: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:638:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:675:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:712:63: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:754:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:785:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:903:40: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:907:40: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:911:41: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:916:40: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:920:41: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:924:38: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:960:31: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _common.pyx:1002:32: Exception check will always require the GIL to be acquired. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
      performance hint: _generator.pyx:707:41: Exception check will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.
      performance hint: _generator.pyx:736:45: Exception check will always require the GIL to be acquired.
      Possible solutions:
          1. Declare the function as 'noexcept' if you control the definition and you're sure you don't want the function to raise exceptions.
          2. Use an 'int' return type on the function to allow an error code to be returned.

      Error compiling Cython file:
      ------------------------------------------------------------
      ...
              for i in range(1, RK_STATE_LEN):
                  self.rng_state.key[i] = val[i]
              self.rng_state.pos = i

              self._bitgen.state = &self.rng_state
              self._bitgen.next_uint64 = &mt19937_uint64
                                         ^
      ------------------------------------------------------------

      _mt19937.pyx:138:35: Cannot assign type 'uint64_t (*)(void *) except? -1 nogil' to 'uint64_t (*)(void *) noexcept nogil'. Exception values are incompatible. Suggest adding 'noexcept' to type 'uint64_t (void *) except? -1 nogil'.
      Processing numpy/random\_bounded_integers.pxd.in
      Processing numpy/random\bit_generator.pyx
      Processing numpy/random\mtrand.pyx
      Processing numpy/random\_bounded_integers.pyx.in
      Processing numpy/random\_common.pyx
      Processing numpy/random\_generator.pyx
      Processing numpy/random\_mt19937.pyx
      Traceback (most recent call last):
        File "C:\Users\Ophek\AppData\Local\Temp\pip-install-n406nd2p\numpy_6a1cb8d904c14ee5920e5b7a6fef2fa6\tools\cythonize.py", line 235, in <module>
          main()
        File "C:\Users\Ophek\AppData\Local\Temp\pip-install-n406nd2p\numpy_6a1cb8d904c14ee5920e5b7a6fef2fa6\tools\cythonize.py", line 231, in main
          find_process_files(root_dir)
        File "C:\Users\Ophek\AppData\Local\Temp\pip-install-n406nd2p\numpy_6a1cb8d904c14ee5920e5b7a6fef2fa6\tools\cythonize.py", line 222, in find_process_files
          process(root_dir, fromfile, tofile, function, hash_db)
        File "C:\Users\Ophek\AppData\Local\Temp\pip-install-n406nd2p\numpy_6a1cb8d904c14ee5920e5b7a6fef2fa6\tools\cythonize.py", line 188, in process
          processor_function(fromfile, tofile)
        File "C:\Users\Ophek\AppData\Local\Temp\pip-install-n406nd2p\numpy_6a1cb8d904c14ee5920e5b7a6fef2fa6\tools\cythonize.py", line 77, in process_pyx
          subprocess.check_call(
        File "C:\Users\Ophek\AppData\Local\Programs\Python\Python312\Lib\subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['C:\\Users\\Ophek\\AppData\\Local\\Programs\\Python\\Python312\\python.exe', '-m', 'cython', '-3', '--fast-fail', '-o', '_mt19937.c', '_mt19937.pyx']' returned non-zero exit status 1.
      Cythonizing sources
      Traceback (most recent call last):
        File "C:\Users\Ophek\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 353, in <module>
          main()
        File "C:\Users\Ophek\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\Ophek\AppData\Local\Programs\Python\Python312\Lib\site-packages\pip\_vendor\pyproject_hooks\_in_process\_in_process.py", line 149, in prepare_metadata_for_build_wheel
          return hook(metadata_directory, config_settings)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "C:\Users\Ophek\AppData\Local\Temp\pip-build-env-6dc9hmi7\overlay\Lib\site-packages\setuptools\build_meta.py", line 366, in prepare_metadata_for_build_wheel
          self.run_setup()
        File "C:\Users\Ophek\AppData\Local\Temp\pip-build-env-6dc9hmi7\overlay\Lib\site-packages\setuptools\build_meta.py", line 480, in run_setup
          super(_BuildMetaLegacyBackend, self).run_setup(setup_script=setup_script)
        File "C:\Users\Ophek\AppData\Local\Temp\pip-build-env-6dc9hmi7\overlay\Lib\site-packages\setuptools\build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 499, in <module>
        File "<string>", line 479, in setup_package
        File "<string>", line 274, in generate_cython
      RuntimeError: Running cythonize failed!
      [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[vinid]

Hello! thanks for reporting the issue!

let me do some experiments and i'll get back to you hopefully soon!

[j-at-ch]

Hi @poopkinel - a bit late, but if you're still stuck on this then I'd take a couple of steps back. Unfortunately the latest versions of Python may have C compilation-related errors on Windows. I'd try using Python 3.8 to start with cf. #148.

It's best to do this in a venv (or a conda environment) so as not to mess up your base operating system Python distribution.