How to increase frame from 8 to a bigger number

[tarunmis]

I need to increase the number of frames for video processing for finetuning and inferencing. Currently it is 8 I want around 32. What is the best way to do it?

[ParkerLischwe]

Also curious about this. @LinB203, is it possible to increase the number of frames used in a video when using LoRA fine tuning? If so, do I just need to change num_frames in processing_video.py to increase amount of frames used for fine tuning as well as inference? What are the differences I should expect when trying to expand the frame amount for fine tuning vs at inference time?

[weikaih04]

Same, wanna do 16 frames in inferencing

[Fritskee]

Change the number of frames you'd like on this line:
https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72

Then go to either of these 2 lines:
https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93
or
https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101

You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit.

From there on you can continue to select which frames you'd want to provide as input.

[sunyclj]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72

Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101

You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit.

From there on you can continue to select which frames you'd want to provide as input.

直接修改/LanguageBind_Video_merge/config.json中的num_frames报错呢？
报错如下：size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024]).

[sunyclj]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72

Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101

You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit.

From there on you can continue to select which frames you'd want to provide as input.

一段视频按照转场分段，每段进行推理描述，最终输出多段的推理结果，请问这个逻辑在哪里增加比较恰当呢？

[Fritskee]

Change the number of frames you'd like on this line: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L72
Then go to either of these 2 lines: https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L93 or https://github.com/PKU-YuanGroup/Video-LLaVA/blob/main/videollava/model/multimodal_encoder/languagebind/video/processing_video.py#L101
You can get an insight to which frames are selected by checking either of those lines, depending on which back-end you're using (decord or opencv). In case you don't know it, you can put a breakpoint at either line and when you execute in debug mode, the breakpoint that is relevant for you will be hit.
From there on you can continue to select which frames you'd want to provide as input.

一段视频按照转场分段，每段进行推理描述，最终输出多段的推理结果，请问这个逻辑在哪里增加比较恰当呢？

If you spoke English, I'd be better able to help you. It's also not my first language, so it's a good common ground for us. Anyway, from the little bit that I do understand I'll give you a pointer.

size mismatch for vision_model.encoder.layers.23.temporal_embedding: copying a param with shape torch.Size([1, 8, 1024]) from checkpoint, the shape in current model is torch.Size([1, 16, 1024])

What the above error is saying is that the dimension of your video encoder has a size mismatch. This is because the original video encoder is trained on 8 frames, hence it expects a Tensor of 1, 8, 1024. You now want to sample 16 frames instead of 8 frames. This means your tensor (and therefore your model) should accept this shape. In this case, it probably means you first need to adapt the architecture and then perform a retraining of the vision encoder.