Add H.264 decoding by utilizing OpenH264 from Cisco

[torokati44]

This is very much a WIP. Progresses #723. Basically pushing only for visibility. Will partially replace #12539 (first only on desktop, in the future, using WebCodecs, entirely).

AFAIK this is the only legal way (without us or our users having to pay royalties to MPEG LA VIA LA) to do this with reasonable portability on desktop, but IANAL. See this little explainer video: https://vimeo.com/79578794

The code is somewhat based on the openh264-rs crate, see ralfbiedert/openh264-rs#43.

YUV->RGB colorspace conversion is done by our trusty handcrafted code, like for H.263 and VP6[A]. This means that it will always assume BT.601 coefficients, even though it could theoretically, possibly be BT.709 or something like that. But the difference is probably subtle enough anyway.

TODOs, in no particular order:

• Add a configure_decoder (or similar) method to the video backend/decoder traits. Currently the preload method is misused to pass the "extradata" to the decoder.
  • Done.
• Figure out how to not create colliding Indexes for proxied and handled videos in the ExternalBackend.
  • Added an enum to store in the new backend's arena. TODO: Add a warning comment.
• Optimize copying YUV data out, and possibly AVCC->AnnexB format conversion
• Handle downloading and hash-checking of the decoder shared libraries for the current platform/arch.
  • Done. TODO: Needs to do error handling better for proper fallback and hints for the user.
• Add the Cisco license text, display it somewhere in the player.
  • Done. The new backend will write it into the current working directory as OpenH264-license.txt upon creation.
• Add a command line option to disable the decoder (as per Cisco's license terms, I hope this will be enough)
  • Done: --enable-openh264 [true|false]
• Switch to preferences for the enablement toggle
• Figure out how, or if, to test this (command line flag for implicit agreement to fetch the decoder library on every run?)
  • Download on first run without confirmation, even in CI.
  • For tests, this should be opt-in, just like "video" and "renderer" in general.
  • Command line flag will be only for opting out.
• Write up a method for updating OpenH264 versions in the future.
• Check if undoing the "start code emulation prevention" for the decoder config (extradata) is needed, and undo it if so.
  • Left this as a TODO comment for now...
• Look at the presentation time offsets as provided by the FLV demuxer (later the MP4 container), and the fields in the SBufferInfo struct from the decoder.
• General code cleanup, including properly working features to disable this, comments, and error handling.
  • Almost done!
• Testing, testing, testing...
  • There is a visual test added, which is something...

TODOs for later:

• WebCodecs implementation for web. Consider whether MSE plus a hidden <video> element would be a feasible fallback - probably not, but who knows.
  I plan to put this into the same ExternalVideoBackend added here. Also with the "other ideas" below. Spares us from duplicating the proxying logic. The Android backend will have to be different though, as it's somewhere else.

Somewhat less closely related TODOs for later:

• MP4 (F4V) support: video: Add support for playing F4V (MP4) files #14655
• AAC-LC decoding support (already available in Symphonia).

Other ideas:

• Different, specific implementations for different platforms: Video Toolbox on macOS, Microsoft Media Foundation or DXVA[2] on Windows, VA-API (or VDPAU...) on Linux, and MediaCodec on Android. Maybe Vulkan Video anywhere applicable. (Loading FFmpeg or GStreamer on Linux would be iffy due to uncertain licensing of the used decoder.)

[torokati44]

This is what I use to test this at the moment:
f4vplayer_h264_flv.zip

I hope it's okay to upload here. Despite the name, it now loads an FLV file. It's originally from here: https://permadi.com/2010/02/flash-playing-flvf4v-videos-with-script/

Of course, the decoder library still has to be obtained manually at the moment.

[danielhjacobs]

Will partially replace #12539 (first only on desktop, in the future, using WebCodecs, entirely).

The VideoDecoder part of WebCodecs (and many other APIs) are only available in secure contexts, right (they're also not yet available in Firefox but I'm sure that will change)? Will this be the case in insecure contexts too? I suppose browsers are moving away from allowing insecure sites at all.

[torokati44]

not yet available in Firefox but I'm sure that will change)?

Certainly, since it already works if enabled in about:config, it's just not yet stable/complete.

The VideoDecoder part of WebCodecs (and many other APIs) are only available in secure contexts, right

I have no idea. Anyway, we can keep the current solution if VideoDecoder is not available for whatever reason.

[torokati44]

There's now a bit of a back-and-forth between video/external: Add OpenH264 decoder and desktop: Add a preference to enable the OpenH264 decoder regarding license handling, but it doesn't bother even me that much to go through and rebase it out...

[Aaron1011]

Could this be done through a build.rs script, instead of committing the file?

[torokati44]

From a technical PoV, of course.
Then, we'd have to vendor (or perhaps obtain in some other way) the C header in question at build time.
I think the license within it allows vendoring, and the header in itself should not be problematic regarding patents or royalties.
That is a smaller file to check in, at the price of another dependency and a smidge longer build time.
Would you prefer that?

[torokati44]

Could you please weigh in on this one too, @Dinnerbone?

[Dinnerbone]

I don't care enough to block this PR. If someone feels strongly about it, let's follow up in another PR later.

[Dinnerbone]

I appreciate the effort to feature gate this, but this is a compile error if the feature is turned off.

[torokati44]

Good catch, thank you! Added an inner block that is conditionally compiled.

[Dinnerbone]

Let's not use current_dir please... that could be anywhere
Maybe our config storage place for now?
Or absolute worst case, the folder that the ruffle executable is in... but that's not likely to be the same as current_dir

[torokati44]

Maybe our config storage place for now?

Ummh, I'd find it weird to put anything "executable" in there.

Or absolute worst case, the folder that the ruffle executable is in...

That should be fine IMHO.

but that's not likely to be the same as current_dir

Yeah, good point, I'm a bit too Linux-CLI-minded... 😅

[torokati44]

Changed to:

        let filepath = std::env::current_exe()?
            .parent()
            .ok_or("Could not determine Ruffle location.")?
            .join(filename);

This way, we don't need to touch .gitignore either, as the lib will be put into target.

[torokati44]

Some further minor TODOs:

• Remove or inactivate the toggle option in the preferences dialog if the feature is not enabled in the build. DONE.
• The keyframe detection does not handle the cases when there are multiple actual NALUs clumped together in a chunk, and the IDR NALU is preceded by a SEI NALU (which is not that rare). Also the SPS/PPS NALUs could perhaps be considered "keyframes"...? Perhaps not?
• Change the "decoder did not produce a frame yet" from an Err to instead return a Result<Option<...>,...> from decode_frame, and make this case a None,

[Dinnerbone]

Works on mac and windows.
Slow on my mac but profiled on windows and 2/3rds of the time is spent in openh264 - the other third is to_rgba which we can look at offloading more of to gpu later maybe