Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Warnings about weights, temperature, top_p, and embedding layer, but it still works. Should I worry about them? #149

Open
secretlycarl opened this issue May 2, 2024 · 0 comments

Comments

@secretlycarl
Copy link

secretlycarl commented May 2, 2024

I'm working on a script to sort videos using Video-LLaVA. I've edited the provided video inference function and have it mostly working (check it out here if interested), but I still get these warnings before it starts to process the videos.

Some weights of the model checkpoint at LanguageBind/Video-LLaVA-7B were not used when initializing LlavaLlamaForCausalLM:
'model.video_tower.video_tower.encoder.layers.18.layer_norm2.bias', 
[long list]
'model.video_tower.video_tower.encoder.layers.17.temporal_layer_norm1.bias']
- This IS expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing LlavaLlamaForCausalLM from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).

I'm just running the provided model from this repo, LanguageBind/Video-LLaVA-7B.

C:\Users\user\anaconda3\Lib\site-packages\transformers\generation\configuration_utils.py:362: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.9` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.

in the mentioned file, temperature is set 1, not 0.9. I also initialize the model in the script with that value.

  warnings.warn(
C:\Users\user\anaconda3\Lib\site-packages\transformers\generation\configuration_utils.py:367: UserWarning: `do_sample` is set to `False`. However, `top_p` is set to `0.6` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `top_p`. This was detected when initializing the generation config instance, which means the corresponding file may hold incorrect parameterization and should be fixed.

also in that file, top_p is set to 1. I added a line to initialize the model with that set in the script and I still get this warning. Somewhere, temperature and top_p are being changed, or different values are being pulled from an unmentioned file.
EDIT: i found where to edit this but it was sort of hard to get to. see the end of the readme on my repo for more info.

  warnings.warn(
You are resizing the embedding layer without providing a `pad_to_multiple_of` parameter. This means that the new embedding dimension will be 32000. This might induce some performance reduction as *Tensor Cores* will not be available. For more details about this, or help on choosing the correct value for resizing, refer to this guide: https://docs.nvidia.com/deeplearning/performance/dl-performance-matrix-multiplication/index.html#requirements-tc

Don't know what to do with this, the link directs to a chart with some info about tensor cores for an A1000. I'm on an RTX 3060.

Are any of these to be concerned about? I'm happy with my script but if it could be improved with these fixes I want to try.

@secretlycarl secretlycarl changed the title A few errors but it still works. Should I worry about them? Warnings about weights, temperature, top_p, and embedding layer, but it still works. Should I worry about them? May 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant