Add Support for YUV420 and YVU variants.

[samlii]

This adds encoder support for the following planar formats from v4l.

• YUV420 (YU12)
• YVU420 (YV12)

Hopefully its pretty straight forward.

I needed this because scrcpy v4l2 sink (through v4l2loopback) uses the YU12 format on my android phones.

I did notice that I once and a while get -- ERROR [47432.502 ] -- CAP: V4L2 error: grabbed HW buffer=0 is already used. I'm not sure why these happen, however I do think these should probably be Warnings and just continue the loop instead of stopping execution to rebuild the capture. Has this grabbed problem been seen where it didn't resolve itself?

[mdevaev]

I'm not sure why these happen, however I do think these should probably be Warnings and just continue the loop instead of stopping execution to rebuild the capture.

This is a bug in v4l2loopback. It re-gives a buffer that has already been taken earlier, but has not yet been released. This should never happen, otherwise multithreaded frame processing will be impossible.

[mdevaev]

Btw what about HW and M2M encoders? H.264?

[samlii]

This is a bug in v4l2loopback. It re-gives a buffer that has already been taken earlier, but has not yet been released. This should never happen, otherwise multithreaded frame processing will be impossible.

Interesting I see the threads now. I wonder if there would be a workaround if we knew it was a loopback (or at least handle it differently for a loopback device.)

Btw what about HW and M2M encoders? H.264?

As I read it hw doesn't make sense since that is only useful for M/JPEG input streams which this is not. The other 2 I didn't understand how they were working.
I don't see anything being handled specially for the packed YUV formats, so I figured it wasn't something I needed to worry about. Where in the M2M stuff is the conversion actually happening? Why it would be different than YUYV?

[mdevaev]

Interesting I see the threads now. I wonder if there would be a workaround if we knew it was a loopback (or at least handle it differently for a loopback device.)

We don't need workaround here, it should be fixed in v4l2loopback. Don't get me wrong, I don't want to disable protection and limit the number of threads simply because someone else's software is not working properly. Bugs should be fixed, not ignored.

As I read it hw doesn't make sense since that is only useful for M/JPEG input streams which this is not.

This is not quite true. M2M works for RAW formats, this is their main purpose. I use this mainly with UYVY. My main goal is that any format should be supported by any encoder.

You can grep format here: https://github.com/pikvm/ustreamer/blob/master/src/ustreamer/m2m.c

[samlii]

We don't need workaround here, it should be fixed in v4l2loopback. Don't get me wrong, I don't want to disable protection and limit the number of threads simply because someone else's software is not working properly. Bugs should be fixed, not ignored.

I agree it should be fixed upstream, was just postulating whether we could make our experience better until then. It looks like there isn't a good workaround (eventually it still segfaults no matter what I do), so this is a non-issue. I'll see if there is any movement on the v4l2loopback side and try to understand the issue better.

As I read it hw doesn't make sense since that is only useful for M/JPEG input streams which this is not.

This is not quite true. M2M works for RAW formats, this is their main purpose. I use this mainly with UYVY. My main goal is that any format should be supported by any encoder.

You can grep format here: https://github.com/pikvm/ustreamer/blob/master/src/ustreamer/m2m.c

That line was in reference to the encoders/hw/encoder.c code, which explicitly says if it's not jpeg don't use it. As for the M2M interface, I still don't see any checks or things to do for raw formats. It appears the code simply sets the input format to whatever the frame is and hopes that the encoder hardware doesn't fail. I don't have a Pi handy so I can't test the M2M interface as is used in the code, however I have seen references to ffmpeg using the hardware encoder with YUV420 (YU12) based input formats successfully, I don't think this should be any different. Is there a location where I can see what input formats the hardware encoder of the Pi supports?

Sorry not trying to be obtuse here, just trying to understand the M2M stuff and how it is used. My use case is not on a Pi, and I realize that isn't the typical case here, so I need a little guidance when it comes to the Pi. Do you know if there is a decent way to test the M2M stuff if I am not on a Pi?

[mdevaev]

Yes, you are right about hw. I didn't get enough sleep. I don't know how to test M2M without Pi, we probably need something like Nvidia or ATI with the same M2M interface, I'm sure the driver is exists.

Here are all formats:

https://github.com/raspberrypi/linux/blob/rpi-6.6.y/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c

[samlii]

Yes, you are right about hw. I didn't get enough sleep. I don't know how to test M2M without Pi, we probably need something like Nvidia or ATI with the same M2M interface, I'm sure the driver is exists.

I think the problem here is that the M2M available source formats are dependent on the ones that are available to the Pi itself.

https://github.com/raspberrypi/linux/blob/rpi-6.6.y/drivers/staging/vc04_services/bcm2835-codec/bcm2835-v4l2-codec.c

Thanks for this, so according to this source the Pi will support the 420 formats, but not the 410 formats for M2M conversion. I'm okay with removing the 410 formats if you want to try to keep the list of formats to only what can be done in M2M. Just let me know.

I am trying to figure out why the CI build keeps failing. It seems to fail on linting of things that weren't touched in my code. Am I missing something?

[mdevaev]

I'll take a look later, sorry, very busy now.

[samlii]

Hey @mdevaev, just wondering if you had time and if you saw anything else that needed to change?

[mdevaev]

I totally forgot about it ._. Hopefully by the end of this month.