TypeError: forward() missing 1 required positional argument: 'hidden_states'[BUG] ?

[silvacarl2]

Describe the bug
We get this error when trying to convert a fine tuned gpt-j model to gptq:

TypeError: forward() missing 1 required positional argument: 'hidden_states'

Hardware details
A40

Software version
nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

NVIDIA-SMI 535.161.08 Driver Version: 535.161.08 CUDA Version: 12.2

Any ideas?

[Qubitium]

@silvacarl2 Try 1) use latest 4.39.3 (latest) transformer or 4.38.2 (stable) 2) install autogtq from repo using git clone

[silvacarl2]

nope:

python convert-to-gptq.py -m ./gpt-cmd -o ./gpt-cmd-gptq
CUDA extension not installed.
CUDA extension not installed.
/home/silvacarl/.local/lib/python3.8/site-packages/torch/_utils.py:831: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
return self.fget.get(instance, owner)()
Quantizing transformer.h blocks : 0%| | 0/28 [00:00<?, ?it/s]
Traceback (most recent call last):
File "convert-to-gptq.py", line 33, in
quantized_model = quantizer.quantize_model(model, tokenizer)
File "/home/silvacarl/.local/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/silvacarl/.local/lib/python3.8/site-packages/optimum/gptq/quantizer.py", line 505, in quantize_model
block(*layer_inputs[j], **layer_input_kwargs[j])
File "/home/silvacarl/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/home/silvacarl/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
return forward_call(*args, **kwargs)
TypeError: forward() missing 1 required positional argument: 'hidden_states'

will try to build the Docker.

[silvacarl2]

Docker file seems to be out of date? get this error:

$ docker build --no-cache --force-rm -t autogptq .

=> ERROR [9/9] RUN pip install -vvv . 1.9s

[9/9] RUN pip install -vvv .:
0.481 Using pip 24.0 from /root/miniconda3/lib/python3.12/site-packages/pip (python 3.12)
0.481 Non-user install because site-packages writeable
0.520 Created temporary directory: /tmp/pip-build-tracker-bbwhjl64
0.520 Initialized build tracking at /tmp/pip-build-tracker-bbwhjl64
0.520 Created build tracker: /tmp/pip-build-tracker-bbwhjl64
0.520 Entered build tracker: /tmp/pip-build-tracker-bbwhjl64
0.520 Created temporary directory: /tmp/pip-install-w4zqf4xa
0.521 Created temporary directory: /tmp/pip-ephem-wheel-cache-exns5lav
0.551 Processing /AutoGPTQ
0.551 Added file:///AutoGPTQ to build tracker '/tmp/pip-build-tracker-bbwhjl64'
0.551 Running setup.py (path:/AutoGPTQ/setup.py) egg_info for package from file:///AutoGPTQ
0.552 Created temporary directory: /tmp/pip-pip-egg-info-tnc1zga1
0.552 Preparing metadata (setup.py): started
0.552 Running command python setup.py egg_info
1.539 Traceback (most recent call last):
1.540 File "", line 2, in
1.540 File "", line 34, in
1.540 File "/AutoGPTQ/setup.py", line 111, in
1.540 local_arch_list = detect_local_sm_architectures()
1.540 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.540 File "/AutoGPTQ/setup.py", line 68, in detect_local_sm_architectures
1.540 arch_list[-1] += '+PTX'
1.540 ~~~~~~~~~^^^^
1.541 IndexError: list index out of range
1.745 error: subprocess-exited-with-error
1.745
1.745 × python setup.py egg_info did not run successfully.
1.745 │ exit code: 1
1.745 ╰─> See above for output.
1.745
1.745 note: This error originates from a subprocess, and is likely not a problem with pip.
1.746 full command: /root/miniconda3/bin/python -c '
1.746 exec(compile('"'"''"'"''"'"'
1.746 # This is -- a caller that pip uses to run setup.py
1.746 #
1.746 # - It imports setuptools before invoking setup.py, to enable projects that directly
1.746 # import from distutils.core to work with newer packaging standards.
1.746 # - It provides a clear error message when setuptools is not installed.
1.746 # - It sets sys.argv[0] to the underlying setup.py, when invoking setup.py so
1.746 # setuptools doesn'"'"'t think the script is -c. This avoids the following warning:
1.746 # manifest_maker: standard file '"'"'-c'"'"' not found".
1.746 # - It generates a shim setup.py, for handling setup.cfg-only projects.
1.746 import os, sys, tokenize
1.746
1.746 try:
1.746 import setuptools
1.746 except ImportError as error:
1.746 print(
1.746 "ERROR: Can not execute setup.py since setuptools is not available in "
1.746 "the build environment.",
1.746 file=sys.stderr,
1.746 )
1.746 sys.exit(1)
1.746
1.746 file = %r
1.746 sys.argv[0] = file
1.746
1.746 if os.path.exists(file):
1.746 filename = file
1.746 with tokenize.open(file) as f:
1.746 setup_py_code = f.read()
1.746 else:
1.746 filename = ""
1.746 setup_py_code = "from setuptools import setup; setup()"
1.746
1.746 exec(compile(setup_py_code, filename, "exec"))
1.746 '"'"''"'"''"'"' % ('"'"'/AutoGPTQ/setup.py'"'"',), "", "exec"))' egg_info --egg-base /tmp/pip-pip-egg-info-tnc1zga1
1.746 cwd: /AutoGPTQ/
1.747 Preparing metadata (setup.py): finished with status 'error'
1.748 error: metadata-generation-failed
1.748
1.748 × Encountered error while generating package metadata.
1.748 ╰─> See above for output.
1.748
1.748 note: This is an issue with the package mentioned above, not pip.
1.748 hint: See above for details.
1.751 Exception information:
1.751 Traceback (most recent call last):
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/operations/build/metadata_legacy.py", line 64, in generate_metadata
1.751 call_subprocess(
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/utils/subprocess.py", line 224, in call_subprocess
1.751 raise error
1.751 pip._internal.exceptions.InstallationSubprocessError: python setup.py egg_info exited with 1
1.751
1.751 The above exception was the direct cause of the following exception:
1.751
1.751 Traceback (most recent call last):
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/cli/base_command.py", line 180, in exc_logging_wrapper
1.751 status = run_func(*args)
1.751 ^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/cli/req_command.py", line 245, in wrapper
1.751 return func(self, options, args)
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/commands/install.py", line 377, in run
1.751 requirement_set = resolver.resolve(
1.751 ^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 76, in resolve
1.751 collected = self.factory.collect_root_requirements(root_reqs)
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 534, in collect_root_requirements
1.751 reqs = list(
1.751 ^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 490, in _make_requirements_from_install_req
1.751 cand = self._make_base_candidate_from_link(
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 228, in _make_base_candidate_from_link
1.751 self._link_candidate_cache[link] = LinkCandidate(
1.751 ^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 290, in init
1.751 super().init(
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in init
1.751 self.dist = self._prepare()
1.751 ^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 222, in _prepare
1.751 dist = self._prepare_distribution()
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 301, in _prepare_distribution
1.751 return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 525, in prepare_linked_requirement
1.751 return self._prepare_linked_requirement(req, parallel_builds)
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 640, in _prepare_linked_requirement
1.751 dist = _get_prepared_distribution(
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/operations/prepare.py", line 71, in _get_prepared_distribution
1.751 abstract_dist.prepare_distribution_metadata(
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/distributions/sdist.py", line 67, in prepare_distribution_metadata
1.751 self.req.prepare_metadata()
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/req/req_install.py", line 585, in prepare_metadata
1.751 self.metadata_directory = generate_metadata_legacy(
1.751 ^^^^^^^^^^^^^^^^^^^^^^^^^
1.751 File "/root/miniconda3/lib/python3.12/site-packages/pip/_internal/operations/build/metadata_legacy.py", line 71, in generate_metadata
1.751 raise MetadataGenerationFailed(package_details=details) from error
1.751 pip._internal.exceptions.MetadataGenerationFailed: metadata generation failed
1.784 Remote version of pip: 24.0
1.784 Local version of pip: 24.0
1.803 Was pip installed by pip? True
1.803 Removed file:///AutoGPTQ from build tracker '/tmp/pip-build-tracker-bbwhjl64'
1.803 Removed build tracker: '/tmp/pip-build-tracker-bbwhjl64'

---

Dockerfile:28

26 | WORKDIR /AutoGPTQ
27 |
28 | >>> RUN pip install -vvv .

ERROR: failed to solve: process "/bin/sh -c pip install -vvv ." did not complete successfully: exit code: 1