-
Notifications
You must be signed in to change notification settings - Fork 400
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[BUG] TypeError: LlamaRotaryEmbedding.forward() got an unexpected keyword argument 'seq_len' #606
Comments
Thank you for the report @timefliesfang, this is likely due to a breaking change in Transformers following https://github.com/huggingface/transformers/releases/tag/v4.38.0. I'll try to fix, but as a temporary workaround, can you try to use transformers==4.37.2? |
Yes, it works now. Thank you for your response:) |
@fxmarty Hi, I'm maintaining a template on runpod with text-generation-webui. A few hours ago the template broke again and I had to fix it by upgrading gradio and transformers. Do you have any idea on when this issue will be fixed? |
Describe the bug
Encounter this when running genreration_speed.py. Input model is Llama-2-7b-chat-gptq. Complete error log:
Traceback (most recent call last):
File "AutoGPTQ/examples/benchmark/generation_speed.py", line 326, in
main()
File "AutoGPTQ/examples/benchmark/generation_speed.py", line 316, in main
benchmark_generation_speed(model, tokenizer, examples, generation_config)
File "AutoGPTQ/examples/benchmark/generation_speed.py", line 197, in benchmark_generation_speed
outputs_ids = model.generate(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/auto_gptq/modeling/_base.py", line 532, in generate
return self.model.generate(**kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/generation/utils.py", line 1527, in generate
result = self._greedy_search(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/generation/utils.py", line 2411, in _greedy_search
outputs = self(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1196, in forward
outputs = self.model(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 1016, in forward
layer_outputs = decoder_layer(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py", line 739, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/auto_gptq/nn_modules/fused_llama_attn.py", line 76, in forward
cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "venv/auto-gptq-cuda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
TypeError: LlamaRotaryEmbedding.forward() got an unexpected keyword argument 'seq_len'
Hardware details
Information about CPU and GPU, such as RAM, number, etc.
Nvidia V100
Software version
Version of relevant software such as operation system, cuda toolkit, python, auto-gptq, pytorch, transformers, accelerate, etc.
CUDA: 12.3 , Pytorch: 2.2.1, transformers: 4.39.1, accelerate: 0.28.0
The text was updated successfully, but these errors were encountered: