Native Hybrid MP4 Muxer

[derrod]

Description

Adds a native MP4 muxer/output to OBS.

Also adds a buffered file serializer, based on ffmpeg-mux I/O buffer implementation and seeking support for the existing array serializer

Currently supported Features:

• H.264 / HEVC / AV1 video including HDR
• AAC/Opus/FLAC/ALAC codec + uncompressed PCM audio
• Multi-channel audio (5.1 etc.)
• Multi-track audio/video (video untested)
• Extended metadata such as track and encoder names
• File splitting (automatic and manual)
• Chapter markers (requires finalization to run)

Coming later in separte PRs:

• MOV flavour
• ProRes

Motivation and Context

The goals of this is to fix the drawbacks that resulted in fragmented MP4 no longer being the default (#10482).

To address those muxer creates something I call - for the lack of a better term - "Hybrid MP4":

• While writing data to disk the MP4 is written as if it were fragmented, with all the resiliency that is to be expected of that
• When stopping the output the file is finalised like a traditional MP4 in a sort of "soft remux", effectively turning the file into a regular MP4

This gives us the best of both worlds, a resilient and recoverable output file with great compatibility.

Technical description

ISO-BMFF/MP4/QuickTime use a common format for data that is pretty simple at it's heart.
The file is made up of boxes ("atoms" in Apple terminology) that have a size and four letter name, e.g. moov.
When a file is written most of the audio and video data will live in data boxes (mdat) which do not have a defined structure. To be able to read and decode a file it is necessary to parse the movie boxe(s) (moov/moof).

Generally speaking, a standard MP4 file has three boxes:

• ftyp Header
• moov Movie box
• mdat Audio/Video Data

Since the contents of the moov box depend on the data in mdat it is often written at the end, rather than the beginning. But this also means that if a file is incomplete, e.g. due to a crash or medium error, it generally cannot be read, as the mdat data by itself does not have any structure or meaning.

In order to allow for streaming of MP4 files while they are being written some additions were made in form of the moof atom, which allows the leading moov to be incomplete and not contain any information about the a/v data. Instead, it only contains the bare minimum of information necessary to setup a decoder. The data is then split into fragments, each of which starts with a moof box, that can be decoded by combining the data from the moov box with the moof box.

A fragmented MP4 file will thusly look like so:
- ftyp Header
- moov Movie Box (without sample information)
- moof Movie Fragment Box
- mdat Fragment Audio/Video data
- moof Movie Fragment Box
- mdat Fragment Audio/Video Data
- ...

While this format is great for streaming and resilience (it does not need a final moov to be written to be readable), it has significant downsides for casual users. For once, playback requires reading all moof boxes to get an accurate duration, which can take a long time on HDDs or network drivers, and some software straight up doesn't support this. While fragmented files can be turned into regular ones, this also requires double the space and can take a while depending on how fast the medium is.

This is where this "hybrid MP4" approach comes in: It can turn a fragmented MP4 file into a regular one by merely writing 44 or 52 bytes (depending on video data size) at the start!

The hybrid MP4 file structure is as follows:
- ftyp Header
- free Placeholder
- moov Movie Box (without sample information)
- moof Movie Fragment Box
- moof Movie Fragment Box
- mdat Fragment Audio/Video data
- moof Movie Fragment Box
- mdat Fragment Audio/Video Data
- ...
- moov Movie Box (with sample information)

Now in order to make this file appear as though it is a regular MP4 file all we need to do is overwrite the free box's size and name so that it becomes an mdat box that spans the entire file up to our final moov box, thus making the file appear as such:

• ftyp Header
• mdat Audio/Video Data
• moov Movie box

We have essentially just hidden the fragmented structure of the file by turning the entire thing into one giant data box that the muxer will not try to parse. Simple as that!

Or to put it into layman's terms:

Future considerations

By having a custom muxer we are more flexible with how we write data to disk, and could adapt new container or codec features more quickly in the future.

This also lays groundwork for upcoming protocols such as Media over QUIC which will likely make use of CMAF which is a subset of ISO BMFF/MP4.

How Has This Been Tested?

Recorded more videos and spent more time in hex editors than I care to admit.

Types of changes

• New feature (non-breaking change which adds functionality)
• Tweak (non-breaking change to improve existing functionality)

Checklist:

• My code has been run through clang-format.
• I have read the contributing document.
• My code is not on the master branch.
• The code has been tested.
• All commit messages are properly formatted and commits squashed where appropriate.
• I have included updates to all appropriate documentation.

[PatTheMav]

Would it make sense to actually make this the new default MP4 muxer at some point (give that it kinda achieves a "best of both worlds") and offer it that way as a "beta" variant at first (and drop the "Hybrid" part that might just incur additional questions)?

My only real concern is if we offer it (even flagged as "experimental" without multichannel and multi-track audio support (HDR metadata is "should-have" because of HEVC support) because fine print is very often not read and users might get burned by their recordings missing those things.

Which would mean the scope should be:

• Add it as a proper "new" MP4 muxer that should become default in the future
• Mark it as a "Beta" opt-in feature
• Ensure at least multichannel+multitrack audio works
• Maybe make it H264/AV1 only at first to prevent HEVC HDR recordings until that is implemented

[derrod]

Would it make sense to actually make this the new default MP4 muxer at some point (give that it kinda achieves a "best of both worlds") and offer it that way as a "beta" variant at first (and drop the "Hybrid" part that might just incur additional questions)?

That's my mid-term goal, at least for Windows/Linux, for macOS I'd want to add support for the MOV specifics necessary to support ProRes first.

My only real concern is if we offer it (even flagged as "experimental" without multichannel and multi-track audio support (HDR metadata is "should-have" because of HEVC support) because fine print is very often not read and users might get burned by their recordings missing those things.

HDR support should be relatively easy to add, I just didn't get around to it yet. Afaik it just requries writing two additional boxes of metadata (clli/mdsv). Right now it should already mostly work, but the missing metadata could result in things not being decoded correctly if the user uses non-default white point/max lightness levels.

Which would mean the scope should be:

• Add it as a proper "new" MP4 muxer that should become default in the future
• Mark it as a "Beta" opt-in feature
• Ensure at least multichannel+multitrack audio works
• Maybe make it H264/AV1 only at first to prevent HEVC HDR recordings until that is implemented

Pretty much my plan.

[ImAciidz]

Tested the feature out, encountered some problems:

The stats panel doesn't show anything for total data output/bitrate/'disk full in...'/etc while recording. Multitrack audio doesn't work, only the first track is recorded.

I did a ~1h44m recording, however the duration is not showing in file explorer/it has the usual fMP4 inconveniences (+ visual playback is garbled whenever I seek in mpv or mpc-hc until I reach the next keyframe in the recording). Looked at the log and I can see that right when I stopped the recording, it says this: 15:57:33.161: Error writing to 'D:/Recordings/2024/04-April/ALT-2024-04-27 14-13-40.mp4', No error
So I'm guessing in this case that something went wrong with the 'soft remux'?

After this I experimented a bit, with some recordings it comes out completely fine, the hybrid container seems to work as expected. Others, however, will at random get the same error text in the log mid-recording (at which point nothing gets recorded anymore, and the resulting file will only play (with the same issues described in the previous paragraph) up to the point of the error). Example log here. I start recording, 30s later the error happens, ~46s later I stop recording, and there is only 30s of video to watch in the resulting file.

Let me know if providing the raw recordings is of any use/help, feel to ping me on Discord (@/aciidz) if it's more convenient or something (I'd be much quicker to respond there than here, at the very least). Can also go for a minimal repro on a completely clean portable build with none of my plugins/scenes/etc if that's desired as well, though I can't imagine how what I currently have would be causing the bugs.

[derrod]

Multi-track audio wasn't working because I forgot the OBS_OUTPUT_MULTI_TRACK_AUDIO flag, I fixed that and it seems to work as expected.

The write error is very odd, I wonder if the error isn't being logged correctly, will have to see if I can reproduce. For now I changed the logging a bit and also made sure the output will stop with an error when the issue occurs.

The other issue (garbled video until you get to a keyframe) sounds like perhaps some file offsets aren't being calculated correctly, will have to investigate.

Finally, output bitrate/bytes written stats were not visible because I forgot to add .get_total_bytes but that's now also fixed.

Edit: Forgot to mention I also added support for the HDR metadata boxes so now that should work fine as well. In diving into channel layouts I discovered that FFmpeg doesn't bother writing those for non-PCM codecs, presumably because they're self-describing. But that will require a bit more research.

[derrod]

I did a ~1h44m recording, however the duration is not showing in file explorer/it has the usual fMP4 inconveniences (+ visual playback is garbled whenever I seek in mpv or mpc-hc until I reach the next keyframe in the recording).

Okay I found what's causing this, the large size (>4 GiB) field wasn't written correctly. If you open the file in a hex editor and search for the first mdat and replace the four bytes directly before it with 00 00 00 01 it should work fine!

This is now also fixed.

Edit: Seems there might still be an issue with fragmentation not always happening on keyframes, will continue to investigate.

[derrod]

Okay the keyframe issue should be fixed as well now, it was the result of the sample flags in fragments not being correct due to a little endiannes mistake. A soft-remuxed file should not be affected as keyframe indices in the full moov were correct.

[ImAciidz]

There's a typo in the mp4_output_actual_stop log message.

Tried the latest build, worked fine with a handful of very short recordings, but I have a new problem. Did a ~18.5 minute recording where the duration isn't showing in file explorer, but it opens quickly (like a normal mp4) in mpc-hc/mpv/mediainfo/etc. However when I tried to start another recording, nothing happened (aside from the Start Recording button getting highlighted until I clicked it again), trying to exit OBS just results in an indefinite hang. Link to the log where this happened (note that I successfully did a bunch of shorter recordings before the last one that broke).

After this I decided to just setup a more minimal repro environment, unfortunately I kinda failed at doing that because I overwrote my install of the NVENC refactor build, so the new obs-nvenc plugin was there/loaded/etc, but I still had the old/expected encoder selected (I just copied basic.ini and recordEncoder.json from my usual OBS over to this one, otherwise everything else should be fresh)...so I hope that had minimal influence lol. Anyways, I started a recording and let it go for ~17.5 minutes, stopped the recording, same issue happened again (file plays fine, but no duration in file explorer, couldn't start another recording, and OBS hung when I tried to close it). Log

[derrod]

Mh I wonder if that's the same deadlock with the threaded writer that #10597 is supposed to fix, but I applied that patch here, so maybe that won't quite fix it :/

This also affects the regular FFmpeg muxer output, but the writer is a separate process entirely there, so it deadlocking doesn't break OBS itself. But can result in fun things like you discovering several hours or even days later (if you don't reboot) that one of your video files is still locked by a stray obs-ffmpeg-mux.exe 😅

I'll add some more logging to see if that is the case for you if I can't reproduce locally.

Edit: So far haven't been able to reproduce it :/ Hopefully the additional logging will provide some clarity on the matter.

Edit 2: I was able to reproduce it, it's not a deadlock but a busy loop. Seems like somewhere the muxer sends a write with a size of 0 which throws the writer for a loop (pun intended). Simply returning immediately if the write is 0 bytes seems like the fix here.

[derrod]

I've finally figured out what is happening and it is indeed my fault for not looking at the buffered writer from the ffmpeg muxer with enough thought. It requires any writes to be <= CHUNK_SIZE and otherwise will fail. I'll have to go over this tomorrow, it's way too late for debugging this today...

[derrod]

Updated again, now with support for file splitting (automatic or manual).

With that it's almost at feature parity with the current (fragmented) MP4 output, the only thing missing afaik is support for the Opus audio codec.

[pkviet]

I made a check for the most common surround layouts 2.1 5.1 & 7.1 for aac opus alac flac and apart from an issue with 7.1 alac, it's all good.
I'll PR a fix for alac 7.1

[pkviet]

LGTM regarding surround sound.

[derrod]

With yesterday's updates Opus audio is now also supported, meaning we've reached feature parity with the existing MP4 output via the ffmpeg muxer. I've now updated the UI part so it's available in simple mode and changed the description slightly.

With that my plan would be the following (with no specific timeline):

0. Come up with a better name than "Hybrid MP4" that hopefully won't confuse users
1. Add the option labelled as a "BETA" in an upcoming OBS release
2. Gather feedback, fix edge cases that may come up
3. Remove the fragmented MP4 option and migrate users to the new muxer
4. Repeat 1-3 with MOV and ProRes support
5. Make the native muxer the default

[DeeDeeG]

I hope this isn't distracting for me to post this, but I tried it and it worked for me.

I have this sample video I recorded on Windows 10 (using the PR's CI binary), and it's playing back okay for me in Windows 10's default player, VLC and mpv (on Windows 10 & macOS 10.15), Quick Look (spacebar in Finder) and QuickTime Player (macOS). As well as in Firefox (Windows 10 and macOS 10.15). It has the duration and basic metadata in Windows' file Explorer, which is nice.

Sample video file made using the custom muxer (click to expand):

View_of_the_Earth_from_the_ISS_Custom_Muxer_MP4.mp4

Re-encode of some public domain NASA footage from here: https://commons.wikimedia.org/wiki/File:View_of_the_Earth_from_the_ISS_(8K).webm

If there's anything else basic you want tested, I can try to do so. I can see if a relative can open it in their Adobe suite if I get a chance as well. But it seems to be a "normal MP4" for most intents/purposes, I suppose. Thanks for working on this!

EDIT to add: I found ImHex on Windows to be helpful as a hex editor, as it understands MP4 structure and can color-highlight and deal with the different data sections accordingly. Perhaps the Ubuntu version of ImHex also does this. (The macOS version of ImHex doesn't do the special MP4 parsing, I don't think.)

[derrod]

Another week another update:

• Rebased so that libobs: Fix obs_parse_avc_header missing high profile parameters #10657 is included to fix the AVC decoder configuration record
• Added support for negative CTS
  • This removes the requirement for an edit list to skip b-frame delay
  • Also required for CMAF compliance (e.g. for MoQ)
  • Currently not the default as it requires support for a higher version of the spec than baseline MP4 (iso4)
    • Will probably make it the default once I've checked if common media players and NLEs support it
• Only write ctts box if a dts-pts offset exists
• Fixed custom option parsing

Custom options currently are:

• skip_soft_remux=1 to skip the finalisation (mainly for testing/debugging)
• write_encoder_info=1 to add the current encoder settings to each track (in JSON format)
• use_metadata_tags=1 to use arbitrary key-value pairs for metadata instead of fixed iTunes keys
• use_negative_cts=1 to use negative CTS instead of edit lists to skip b-frame delay