403 Server Error: Downloading en_core_web_sm fails due to "Compatibility table not found for Spacy v3.7.5"

[mkh1991]

How to reproduce the behaviour

Not sure if it's easily reproducible (it isn't even for me consistently), but during docker build, downloading the en_core_web_sm package fails with a 403 Server error, on occasion. With the --no-cache option during docker build, it still fails occasionally, and retrying the build often leads to a success, but not always. This has only started happening since sometime early this week or late last week, there's been no code change at all for the build or for requirements, and I've never faced this issue prior to now. spaCy claims it can't find the compatibility table, but clearly it can when it doesn't fail.

Failing command

python -m spacy download en_core_web_sm

Error message

81.32 ✘ Server error (403)
81.32 Couldn't fetch compatibility table. Please find a package for your spaCy
81.32 installation (v3.7.5), and download it manually. For more details, see the
81.32 documentation: https://spacy.io/usage/models

Your Environment

• Operating System: Ubuntu 24.04 LTS
• Python Version Used: 3.11
• spaCy Version Used: 3.7.5
• Environment Information: Conda environment within Docker, spaCy is downloaded with pip install -r requirements.txt

[Lixiyao-meow]

Same issue here

[lgnashold]

Same issue here

[timleslie]

Ditto, but with en_core_web_trf.

[Shayfiz]

I am having the same issue but with SpaCy v3.7.4. Did anyone have any luck with downgrading the version ?

[mkh1991]

Fix

Figured out a fix. Use the following requirements in requirements.txt:

spacy==3.7.5
en_core_web_sm @ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl

Use line 2 above instead of downloading it via python -m spacy download en_core_web_sm. Why 3.7.1 for the model? Find out below!

Root Cause

The direct download link throws a 404 for all SpaCy versions from 3.7.2 to 3.7.5: e.g. https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.2/en_core_web_sm-3.7.2-py3-none-any.whl

I tried downgrading to 3.7.1, and installing directly with en_core_web_sm@ https://github.com/explosion/spacy-models/releases/download/en_core_web_sm-3.7.1/en_core_web_sm-3.7.1-py3-none-any.whl in requirements.txt and got this rather interesting error message:

23.53 INFO: pip is looking at multiple versions of en-core-web-sm to determine which version is compatible with other requirements. This could take a while.
23.53
23.53 The conflict is caused by:
23.53     The user requested spacy==3.7.1
23.53     en-core-web-sm 3.7.1 depends on spacy<3.8.0 and >=3.7.2

Following this odd recommendation (using spacy==3.7.5 but en_core_web_sm==3.7.1) fixed it though.

[darwin403]

Smae issue here. The simple quickstart from official spacy docs doesn't seem to work:

https://spacy.io/usage/models#quickstart

[cplonski20]

This is affecting me as well. Is this being worked on?

[mkh1991]

@cplonski20 figured out a workaround for it, see my comment above.

[mkh1991]

Update 2: this was solved for me using an x86 machine, didn't find a way for it to work on linux-aarch64

Update: this is no longer working, installation fails at the spacy=3.7.5 step now, citing an issue with cymem and murmurhash.

Logs

45.27   Building wheel for cymem (pyproject.toml): started
45.27   Building wheel for cymem (pyproject.toml): finished with status 'error'
45.27   Building wheel for murmurhash (pyproject.toml): started
45.27   Building wheel for murmurhash (pyproject.toml): finished with status 'error'
45.27 Successfully built langdetect myproject
45.27 Failed to build cymem murmurhash
45.27 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable.It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
45.27   error: subprocess-exited-with-error
45.27
45.27   × Building wheel for cymem (pyproject.toml) did not run successfully.
45.27   │ exit code: 1
45.27   ╰─> [17 lines of output]
45.27       running bdist_wheel
45.27       running build
45.27       running build_py
45.27       creating build/lib.linux-aarch64-cpython-311/cymem
45.27       copying cymem/about.py -> build/lib.linux-aarch64-cpython-311/cymem
45.27       copying cymem/__init__.py -> build/lib.linux-aarch64-cpython-311/cymem
45.27       creating build/lib.linux-aarch64-cpython-311/cymem/tests
45.27       copying cymem/tests/test_import.py -> build/lib.linux-aarch64-cpython-311/cymem/tests
45.27       copying cymem/tests/__init__.py -> build/lib.linux-aarch64-cpython-311/cymem/tests
45.27       copying cymem/cymem.pyx -> build/lib.linux-aarch64-cpython-311/cymem
45.27       copying cymem/cymem.pxd -> build/lib.linux-aarch64-cpython-311/cymem
45.27       copying cymem/__init__.pxd -> build/lib.linux-aarch64-cpython-311/cymem
45.27       running build_ext
45.27       building 'cymem.cymem' extension
45.27       creating build/temp.linux-aarch64-cpython-311/cymem
45.27       g++ -pthread -B /opt/conda/envs/myproject/compiler_compat -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/envs/myproject/include -fPIC -O3 -isystem /opt/conda/envs/myproject/include -fPIC -I/opt/conda/envs/myproject/include/python3.11 -I/opt/conda/envs/myproject/include/python3.11 -c cymem/cymem.cpp -o build/temp.linux-aarch64-cpython-311/cymem/cymem.o -O3 -Wno-strict-prototypes -Wno-unused-function
45.27       error: command 'g++' failed: No such file or directory
45.27       [end of output]
45.27
45.27   note: This error originates from a subprocess, and is likely not a problem with pip.
45.27   ERROR: Failed building wheel for cymem
45.27   error: subprocess-exited-with-error
45.27
45.27   × Building wheel for murmurhash (pyproject.toml) did not run successfully.
45.27   │ exit code: 1
45.27   ╰─> [21 lines of output]
45.27       running bdist_wheel
45.27       running build
45.27       running build_py
45.27       creating build/lib.linux-aarch64-cpython-311/murmurhash
45.27       copying murmurhash/about.py -> build/lib.linux-aarch64-cpython-311/murmurhash
45.27       copying murmurhash/__init__.py -> build/lib.linux-aarch64-cpython-311/murmurhash
45.27       creating build/lib.linux-aarch64-cpython-311/murmurhash/tests
45.27       copying murmurhash/tests/test_import.py -> build/lib.linux-aarch64-cpython-311/murmurhash/tests
45.27       copying murmurhash/tests/test_hash.py -> build/lib.linux-aarch64-cpython-311/murmurhash/tests
45.27       copying murmurhash/tests/__init__.py -> build/lib.linux-aarch64-cpython-311/murmurhash/tests
45.27       copying murmurhash/mrmr.pyx -> build/lib.linux-aarch64-cpython-311/murmurhash
45.27       copying murmurhash/mrmr.pxd -> build/lib.linux-aarch64-cpython-311/murmurhash
45.27       copying murmurhash/__init__.pxd -> build/lib.linux-aarch64-cpython-311/murmurhash
45.27       creating build/lib.linux-aarch64-cpython-311/murmurhash/include/murmurhash
45.27       copying murmurhash/include/murmurhash/MurmurHash3.h -> build/lib.linux-aarch64-cpython-311/murmurhash/include/murmurhash
45.27       copying murmurhash/include/murmurhash/MurmurHash2.h -> build/lib.linux-aarch64-cpython-311/murmurhash/include/murmurhash
45.27       running build_ext
45.27       building 'murmurhash.mrmr' extension
45.27       creating build/temp.linux-aarch64-cpython-311/murmurhash
45.27       g++ -pthread -B /opt/conda/envs/myproject/compiler_compat -Wsign-compare -DNDEBUG -fwrapv -O3 -Wall -fPIC -O3 -isystem /opt/conda/envs/myproject/include -fPIC -O3 -isystem /opt/conda/envs/myproject/include -fPIC -I/opt/conda/envs/myproject/include/python3.11 -I/tmp/pip-install-iaxt4tat/murmurhash_db44aaf71af34742aabaf0eae4f05f56/murmurhash/include -I/opt/conda/envs/myproject/include/python3.11 -c murmurhash/MurmurHash2.cpp -o build/temp.linux-aarch64-cpython-311/murmurhash/MurmurHash2.o -O3 -Wno-strict-prototypes -Wno-unused-function
45.27       error: command 'g++' failed: No such file or directory
45.27       [end of output]
45.27
45.27   note: This error originates from a subprocess, and is likely not a problem with pip.
45.27   ERROR: Failed building wheel for murmurhash
45.27 ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (cymem, murmurhash)
45.27 make: *** [Makefile:24: requirements] Error 1
45.27
45.27 ERROR conda.cli.main_run:execute(125): `conda run /bin/bash -c make requirements` failed. (See above for error)
45.27
------