Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Trouble installing MMF on Colab #1322

Open
catherinening opened this issue Nov 9, 2023 · 1 comment
Open

Trouble installing MMF on Colab #1322

catherinening opened this issue Nov 9, 2023 · 1 comment

Comments

@catherinening
Copy link

Hi, I have been able to successfully install mmf on my local machine (an intel Mac), but it does not support a GPU version of pytorch. Thus, I am trying to get mmf running on Colab, following the same process I did to install mmf locally, but I keep running into the same error. ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects. I was able to get past this error locally using the approach I summarized below, but not on Colab. Does anyone have a solution? Do I need to make further changes to requirements.txt?

Steps to reproduce the error:

%cd '/content/drive/MyDrive/[project_dir]'
# install fastText locally
!git clone https://github.com/facebookresearch/fastText.git
!cd fastText
!make
%cd '/content/drive/MyDrive/[project_dir]/fastText' # install fastText
!sudo python setup.py install

# install tokenizers, lmdb, transformers
%cd '/content/drive/MyDrive/[project_dir]'
!pip install tokenizers
!pip install lmdb
!pip install transformers

# clone mmf repo
!git clone https://github.com/facebookresearch/mmf.git mmf

Then, I updated the requirements.txt file from the cloned repo, editing the fastText, lmdb, tokenizers, and transformers versions, and changing the pycocotools version to 2.0.7

Then, I run:

!pip install -r requirements.txt
!pip install --editable .

My full output, including the error message:

/content/drive/MyDrive/project/CS7643_project/mmf
Collecting torch==1.11.0 (from -r requirements.txt (line 1))
  Using cached torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl (750.6 MB)
Collecting torchaudio==0.11.0 (from -r requirements.txt (line 2))
  Using cached torchaudio-0.11.0-cp310-cp310-manylinux1_x86_64.whl (2.9 MB)
Collecting torchvision==0.12.0 (from -r requirements.txt (line 3))
  Using cached torchvision-0.12.0-cp310-cp310-manylinux1_x86_64.whl (21.0 MB)
Collecting numpy<=1.21.4,>=1.16.6 (from -r requirements.txt (line 4))
  Using cached numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
Collecting tqdm<4.50.0,>=4.43.0 (from -r requirements.txt (line 5))
  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
Collecting torchtext==0.12.0 (from -r requirements.txt (line 6))
  Using cached torchtext-0.12.0-cp310-cp310-manylinux1_x86_64.whl (10.4 MB)
Collecting GitPython==3.1.30 (from -r requirements.txt (line 7))
  Using cached GitPython-3.1.30-py3-none-any.whl (184 kB)
Collecting requests==2.23.0 (from -r requirements.txt (line 8))
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: fasttext==0.9.2 in /usr/local/lib/python3.10/dist-packages/fasttext-0.9.2-py3.10-linux-x86_64.egg (from -r requirements.txt (line 9)) (0.9.2)
Collecting nltk==3.6.6 (from -r requirements.txt (line 10))
  Using cached nltk-3.6.6-py3-none-any.whl (1.5 MB)
Collecting editdistance==0.5.3 (from -r requirements.txt (line 11))
  Using cached editdistance-0.5.3-cp310-cp310-linux_x86_64.whl
Collecting transformers<=4.10.1,>=3.4.0 (from -r requirements.txt (line 12))
  Using cached transformers-4.10.1-py3-none-any.whl (2.8 MB)
Collecting sklearn==0.0 (from -r requirements.txt (line 13))
  Using cached sklearn-0.0-py2.py3-none-any.whl
Collecting omegaconf<=2.1,>=2.0.6 (from -r requirements.txt (line 14))
  Using cached omegaconf-2.1.0-py3-none-any.whl (74 kB)
Requirement already satisfied: lmdb==1.4.1 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 15)) (1.4.1)
Collecting termcolor==1.1.0 (from -r requirements.txt (line 16))
  Using cached termcolor-1.1.0-py3-none-any.whl
Collecting iopath==0.1.8 (from -r requirements.txt (line 17))
  Using cached iopath-0.1.8-py3-none-any.whl (19 kB)
Collecting datasets==1.2.1 (from -r requirements.txt (line 18))
  Using cached datasets-1.2.1-py3-none-any.whl (159 kB)
Collecting matplotlib==3.3.4 (from -r requirements.txt (line 19))
  Using cached matplotlib-3.3.4-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: pycocotools==2.0.7 in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 20)) (2.0.7)
Collecting ftfy==5.8 (from -r requirements.txt (line 21))
  Using cached ftfy-5.8-py3-none-any.whl
Collecting pytorch-lightning==1.6.0 (from -r requirements.txt (line 22))
  Using cached pytorch_lightning-1.6.0-py3-none-any.whl (582 kB)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from -r requirements.txt (line 23)) (5.9.5)
Collecting pillow==9.3.0 (from -r requirements.txt (line 24))
  Using cached Pillow-9.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.3 MB)
Collecting sentencepiece (from -r requirements.txt (line 25))
  Using cached sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch==1.11.0->-r requirements.txt (line 1)) (4.5.0)
Collecting gitdb<5,>=4.0.1 (from GitPython==3.1.30->-r requirements.txt (line 7))
  Using cached gitdb-4.0.11-py3-none-any.whl (62 kB)
Collecting chardet<4,>=3.0.2 (from requests==2.23.0->-r requirements.txt (line 8))
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5 (from requests==2.23.0->-r requirements.txt (line 8))
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests==2.23.0->-r requirements.txt (line 8))
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests==2.23.0->-r requirements.txt (line 8)) (2023.7.22)
Requirement already satisfied: pybind11>=2.2 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->-r requirements.txt (line 9)) (2.11.1)
Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->-r requirements.txt (line 9)) (67.7.2)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->-r requirements.txt (line 10)) (8.1.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->-r requirements.txt (line 10)) (1.3.2)
Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->-r requirements.txt (line 10)) (2023.6.3)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sklearn==0.0->-r requirements.txt (line 13)) (1.2.2)
Collecting portalocker (from iopath==0.1.8->-r requirements.txt (line 17))
  Using cached portalocker-2.8.2-py3-none-any.whl (17 kB)
Requirement already satisfied: pyarrow>=0.17.1 in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->-r requirements.txt (line 18)) (9.0.0)
Collecting dill (from datasets==1.2.1->-r requirements.txt (line 18))
  Using cached dill-0.3.7-py3-none-any.whl (115 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->-r requirements.txt (line 18)) (1.5.3)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->-r requirements.txt (line 18)) (3.4.1)
Collecting multiprocess (from datasets==1.2.1->-r requirements.txt (line 18))
  Using cached multiprocess-0.70.15-py310-none-any.whl (134 kB)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (0.12.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (1.4.5)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (3.1.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->-r requirements.txt (line 19)) (2.8.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from ftfy==5.8->-r requirements.txt (line 21)) (0.2.9)
Requirement already satisfied: PyYAML>=5.4 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (6.0.1)
Requirement already satisfied: fsspec[http]!=2021.06.0,>=2021.05.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2023.6.0)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2.14.1)
Collecting torchmetrics>=0.4.1 (from pytorch-lightning==1.6.0->-r requirements.txt (line 22))
  Using cached torchmetrics-1.2.0-py3-none-any.whl (805 kB)
Collecting pyDeprecate<0.4.0,>=0.3.1 (from pytorch-lightning==1.6.0->-r requirements.txt (line 22))
  Using cached pyDeprecate-0.3.2-py3-none-any.whl (10 kB)
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (23.2)
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12)) (3.13.1)
Requirement already satisfied: huggingface-hub>=0.0.12 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12)) (0.17.3)
Collecting sacremoses (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12))
  Using cached sacremoses-0.1.1-py3-none-any.whl (897 kB)
Collecting tokenizers<0.11,>=0.10.1 (from transformers<=4.10.1,>=3.4.0->-r requirements.txt (line 12))
  Using cached tokenizers-0.10.3.tar.gz (212 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting antlr4-python3-runtime==4.8 (from omegaconf<=2.1,>=2.0.6->-r requirements.txt (line 14))
  Using cached antlr4_python3_runtime-4.8-py3-none-any.whl
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.8.6)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython==3.1.30->-r requirements.txt (line 7))
  Using cached smmap-5.0.1-py3-none-any.whl (24 kB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.1->matplotlib==3.3.4->-r requirements.txt (line 19)) (1.16.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.4.0)
Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.59.2)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2.17.3)
Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.0.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.5.1)
Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.20.3)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.0.1)
Collecting lightning-utilities>=0.8.0 (from torchmetrics>=0.4.1->pytorch-lightning==1.6.0->-r requirements.txt (line 22))
  Using cached lightning_utilities-0.9.0-py3-none-any.whl (23 kB)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets==1.2.1->-r requirements.txt (line 18)) (2023.3.post1)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->-r requirements.txt (line 13)) (1.11.3)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->-r requirements.txt (line 13)) (3.2.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.3.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (4.0.3)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.3.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (1.3.1)
INFO: pip is looking at multiple versions of scipy to determine which version is compatible with other requirements. This could take a while.
Collecting scipy>=1.3.2 (from scikit-learn->sklearn==0.0->-r requirements.txt (line 13))
  Using cached scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.4 MB)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (2.1.3)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (0.5.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->-r requirements.txt (line 22)) (3.2.2)
Building wheels for collected packages: tokenizers
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
Obtaining file:///content/drive/MyDrive/project/CS7643_project/mmf
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Collecting torch==1.11.0 (from mmf==1.0.0rc12)
  Using cached torch-1.11.0-cp310-cp310-manylinux1_x86_64.whl (750.6 MB)
Collecting torchaudio==0.11.0 (from mmf==1.0.0rc12)
  Using cached torchaudio-0.11.0-cp310-cp310-manylinux1_x86_64.whl (2.9 MB)
Collecting torchvision==0.12.0 (from mmf==1.0.0rc12)
  Using cached torchvision-0.12.0-cp310-cp310-manylinux1_x86_64.whl (21.0 MB)
Collecting numpy<=1.21.4,>=1.16.6 (from mmf==1.0.0rc12)
  Using cached numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB)
Collecting tqdm<4.50.0,>=4.43.0 (from mmf==1.0.0rc12)
  Using cached tqdm-4.49.0-py2.py3-none-any.whl (69 kB)
Collecting torchtext==0.12.0 (from mmf==1.0.0rc12)
  Using cached torchtext-0.12.0-cp310-cp310-manylinux1_x86_64.whl (10.4 MB)
Collecting GitPython==3.1.30 (from mmf==1.0.0rc12)
  Using cached GitPython-3.1.30-py3-none-any.whl (184 kB)
Collecting requests==2.23.0 (from mmf==1.0.0rc12)
  Using cached requests-2.23.0-py2.py3-none-any.whl (58 kB)
Requirement already satisfied: fasttext==0.9.2 in /usr/local/lib/python3.10/dist-packages/fasttext-0.9.2-py3.10-linux-x86_64.egg (from mmf==1.0.0rc12) (0.9.2)
Collecting nltk==3.6.6 (from mmf==1.0.0rc12)
  Using cached nltk-3.6.6-py3-none-any.whl (1.5 MB)
Collecting editdistance==0.5.3 (from mmf==1.0.0rc12)
  Using cached editdistance-0.5.3-cp310-cp310-linux_x86_64.whl
Collecting transformers<=4.10.1,>=3.4.0 (from mmf==1.0.0rc12)
  Using cached transformers-4.10.1-py3-none-any.whl (2.8 MB)
Collecting sklearn==0.0 (from mmf==1.0.0rc12)
  Using cached sklearn-0.0-py2.py3-none-any.whl
Collecting omegaconf<=2.1,>=2.0.6 (from mmf==1.0.0rc12)
  Using cached omegaconf-2.1.0-py3-none-any.whl (74 kB)
Requirement already satisfied: lmdb==1.4.1 in /usr/local/lib/python3.10/dist-packages (from mmf==1.0.0rc12) (1.4.1)
Collecting termcolor==1.1.0 (from mmf==1.0.0rc12)
  Using cached termcolor-1.1.0-py3-none-any.whl
Collecting iopath==0.1.8 (from mmf==1.0.0rc12)
  Using cached iopath-0.1.8-py3-none-any.whl (19 kB)
Collecting datasets==1.2.1 (from mmf==1.0.0rc12)
  Using cached datasets-1.2.1-py3-none-any.whl (159 kB)
Collecting matplotlib==3.3.4 (from mmf==1.0.0rc12)
  Using cached matplotlib-3.3.4-cp310-cp310-linux_x86_64.whl
Requirement already satisfied: pycocotools==2.0.7 in /usr/local/lib/python3.10/dist-packages (from mmf==1.0.0rc12) (2.0.7)
Collecting ftfy==5.8 (from mmf==1.0.0rc12)
  Using cached ftfy-5.8-py3-none-any.whl
Collecting pytorch-lightning==1.6.0 (from mmf==1.0.0rc12)
  Using cached pytorch_lightning-1.6.0-py3-none-any.whl (582 kB)
Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from mmf==1.0.0rc12) (5.9.5)
Collecting pillow==9.3.0 (from mmf==1.0.0rc12)
  Using cached Pillow-9.3.0-cp310-cp310-manylinux_2_28_x86_64.whl (3.3 MB)
Collecting sentencepiece (from mmf==1.0.0rc12)
  Using cached sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)
Requirement already satisfied: pyarrow>=0.17.1 in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->mmf==1.0.0rc12) (9.0.0)
Collecting dill (from datasets==1.2.1->mmf==1.0.0rc12)
  Using cached dill-0.3.7-py3-none-any.whl (115 kB)
Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->mmf==1.0.0rc12) (1.5.3)
Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets==1.2.1->mmf==1.0.0rc12) (3.4.1)
Collecting multiprocess (from datasets==1.2.1->mmf==1.0.0rc12)
  Using cached multiprocess-0.70.15-py310-none-any.whl (134 kB)
Requirement already satisfied: pybind11>=2.2 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->mmf==1.0.0rc12) (2.11.1)
Requirement already satisfied: setuptools>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from fasttext==0.9.2->mmf==1.0.0rc12) (67.7.2)
Requirement already satisfied: wcwidth in /usr/local/lib/python3.10/dist-packages (from ftfy==5.8->mmf==1.0.0rc12) (0.2.9)
Collecting gitdb<5,>=4.0.1 (from GitPython==3.1.30->mmf==1.0.0rc12)
  Using cached gitdb-4.0.11-py3-none-any.whl (62 kB)
Collecting portalocker (from iopath==0.1.8->mmf==1.0.0rc12)
  Using cached portalocker-2.8.2-py3-none-any.whl (17 kB)
Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (0.12.1)
Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (1.4.5)
Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.3 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (3.1.1)
Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.10/dist-packages (from matplotlib==3.3.4->mmf==1.0.0rc12) (2.8.2)
Requirement already satisfied: click in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->mmf==1.0.0rc12) (8.1.7)
Requirement already satisfied: joblib in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->mmf==1.0.0rc12) (1.3.2)
Requirement already satisfied: regex>=2021.8.3 in /usr/local/lib/python3.10/dist-packages (from nltk==3.6.6->mmf==1.0.0rc12) (2023.6.3)
Requirement already satisfied: PyYAML>=5.4 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (6.0.1)
Requirement already satisfied: fsspec[http]!=2021.06.0,>=2021.05.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2023.6.0)
Requirement already satisfied: tensorboard>=2.2.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2.14.1)
Collecting torchmetrics>=0.4.1 (from pytorch-lightning==1.6.0->mmf==1.0.0rc12)
  Using cached torchmetrics-1.2.0-py3-none-any.whl (805 kB)
Collecting pyDeprecate<0.4.0,>=0.3.1 (from pytorch-lightning==1.6.0->mmf==1.0.0rc12)
  Using cached pyDeprecate-0.3.2-py3-none-any.whl (10 kB)
Requirement already satisfied: packaging>=17.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (23.2)
Requirement already satisfied: typing-extensions>=4.0.0 in /usr/local/lib/python3.10/dist-packages (from pytorch-lightning==1.6.0->mmf==1.0.0rc12) (4.5.0)
Collecting chardet<4,>=3.0.2 (from requests==2.23.0->mmf==1.0.0rc12)
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting idna<3,>=2.5 (from requests==2.23.0->mmf==1.0.0rc12)
  Using cached idna-2.10-py2.py3-none-any.whl (58 kB)
Collecting urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 (from requests==2.23.0->mmf==1.0.0rc12)
  Using cached urllib3-1.25.11-py2.py3-none-any.whl (127 kB)
Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests==2.23.0->mmf==1.0.0rc12) (2023.7.22)
Requirement already satisfied: scikit-learn in /usr/local/lib/python3.10/dist-packages (from sklearn==0.0->mmf==1.0.0rc12) (1.2.2)
Collecting antlr4-python3-runtime==4.8 (from omegaconf<=2.1,>=2.0.6->mmf==1.0.0rc12)
  Using cached antlr4_python3_runtime-4.8-py3-none-any.whl
Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12) (3.13.1)
Requirement already satisfied: huggingface-hub>=0.0.12 in /usr/local/lib/python3.10/dist-packages (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12) (0.17.3)
Collecting sacremoses (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12)
  Using cached sacremoses-0.1.1-py3-none-any.whl (897 kB)
Collecting tokenizers<0.11,>=0.10.1 (from transformers<=4.10.1,>=3.4.0->mmf==1.0.0rc12)
  Using cached tokenizers-0.10.3.tar.gz (212 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: aiohttp!=4.0.0a0,!=4.0.0a1 in /usr/local/lib/python3.10/dist-packages (from fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.8.6)
Collecting smmap<6,>=3.0.1 (from gitdb<5,>=4.0.1->GitPython==3.1.30->mmf==1.0.0rc12)
  Using cached smmap-5.0.1-py3-none-any.whl (24 kB)
Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.10/dist-packages (from python-dateutil>=2.1->matplotlib==3.3.4->mmf==1.0.0rc12) (1.16.0)
Requirement already satisfied: absl-py>=0.4 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.4.0)
Requirement already satisfied: grpcio>=1.48.2 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.59.2)
Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2.17.3)
Requirement already satisfied: google-auth-oauthlib<1.1,>=0.5 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.0.0)
Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.5.1)
Requirement already satisfied: protobuf>=3.19.6 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.20.3)
Requirement already satisfied: tensorboard-data-server<0.8.0,>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (0.7.2)
Requirement already satisfied: werkzeug>=1.0.1 in /usr/local/lib/python3.10/dist-packages (from tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.0.1)
Collecting lightning-utilities>=0.8.0 (from torchmetrics>=0.4.1->pytorch-lightning==1.6.0->mmf==1.0.0rc12)
  Using cached lightning_utilities-0.9.0-py3-none-any.whl (23 kB)
Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets==1.2.1->mmf==1.0.0rc12) (2023.3.post1)
Requirement already satisfied: scipy>=1.3.2 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->mmf==1.0.0rc12) (1.11.3)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from scikit-learn->sklearn==0.0->mmf==1.0.0rc12) (3.2.0)
Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (23.1.0)
Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.3.2)
Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (6.0.4)
Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (4.0.3)
Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.9.2)
Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.4.0)
Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp!=4.0.0a0,!=4.0.0a1->fsspec[http]!=2021.06.0,>=2021.05.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.3.1)
Requirement already satisfied: cachetools<6.0,>=2.0.0 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (5.3.2)
Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (0.3.0)
Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.10/dist-packages (from google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (4.9)
Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.10/dist-packages (from google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (1.3.1)
INFO: pip is looking at multiple versions of scipy to determine which version is compatible with other requirements. This could take a while.
Collecting scipy>=1.3.2 (from scikit-learn->sklearn==0.0->mmf==1.0.0rc12)
  Using cached scipy-1.11.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.11.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (36.3 MB)
  Using cached scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.4 MB)
Requirement already satisfied: MarkupSafe>=2.1.1 in /usr/local/lib/python3.10/dist-packages (from werkzeug>=1.0.1->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (2.1.3)
Requirement already satisfied: pyasn1<0.6.0,>=0.4.6 in /usr/local/lib/python3.10/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (0.5.0)
Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.10/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<1.1,>=0.5->tensorboard>=2.2.0->pytorch-lightning==1.6.0->mmf==1.0.0rc12) (3.2.2)
Building wheels for collected packages: mmf, tokenizers
  Building editable for mmf (pyproject.toml) ... done
  Created wheel for mmf: filename=mmf-1.0.0rc12-0.editable-cp310-cp310-linux_x86_64.whl size=10777 sha256=69fd1c4e0c8b6f8becbb6ff719bad5c3d359909e8f27534832e8d3f4c00ea5c0
  Stored in directory: /tmp/pip-ephem-wheel-cache-28ewnq3v/wheels/27/dc/f5/d98920883e2d6e3e0a99bb29bc96ae614580aa1be72498e749
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Successfully built mmf
Failed to build tokenizers
ERROR: Could not build wheels for tokenizers, which is required to install pyproject.toml-based projects
@forro1
Copy link

forro1 commented May 6, 2024

Hey,
I met the same error, do you have a solution?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@catherinening @forro1