Receiving H264 RTP stream from Unreal Engine 5 Pixel Streaming Plugin

[ClayJay3]

I need some advice on how I could go about receiving video streams from Unreal Engine 5's Pixel Streaming Plugin. The plugin uses a WebRTC connection to stream video/audio/controls to a web browser client, it also hosts its own signalling server. I need to receiving just the video streams in my C++ codebase for computer vision processing.

To do this, I'm attempting to use libdatachannel's media tracks to get the H264 data and then eventually I'll use FFMPEG to decode the H264 data into something usable by OpenCV.

I am currently able to connection to the signalling server through the websocket and I receive an SDP offer and send back an answer. After this the PeerConnection is made, a datachannel is opened, and I get a track through the PeerConnection->onTrack() callback. So I think I'm close to getting it to work, I just need some advice on how/if I should use the media handler and if it will work with the Pixel Streaming Plugin.

This is my current onTrack() callback.

// Add peer connection track callback. It's only safe to add this callback after the data channel is opened.
            m_pPeerConnection->onTrack(
                [this](std::shared_ptr<rtc::Track> rtcTrack)
                {
                    // Submit logger message.
                    rtc::Description::Media rtcMediaDescription = rtcTrack->description();
                    // Get some information about the track.
                    std::string szMediaType = rtcMediaDescription.type();

                    // Check if the track is a video track.
                    if (szMediaType != "video")
                    {
                        return;
                    }

                    // Set member variable to the video track.
                    rtcVideoTrack1 = rtcTrack;

                    // Create a H264 depacketization handler and rtcp receiving session.
                    rtcH264DepacketizationHandler = std::make_shared<rtc::H264RtpDepacketizer>();
                    // rtcRTCPReceivingSession       = std::make_shared<rtc::RtcpReceivingSession>();
                    // rtcRTPDepacketizer = std::make_shared<rtc::RtpDepacketizer>();
                    // rtcRTCPSrReporter             = std::make_shared<rtc::RtcpSrReporter>(
                    // std::make_shared<rtc::RtpPacketizationConfig>(rtc::SSRC(0), "ZEDFrontRGBAndDepth", 96, rtc::H264RtpPacketizer::defaultClockRate));
                    // rtcRTCPNackResponder = std::make_shared<rtc::RtcpNackResponder>();
                    // For the receiving size of the media track, the pipeline is run from first to last for sending and from last to first for
                    // receiving, which allows you to setup a bidirectional track. Set the media handler for the video track.
                    // rtcH264DepacketizationHandler->addToChain(rtcRTCPSrReporter);
                    // rtcH264DepacketizationHandler->addToChain(rtcRTCPNackResponder);
                    // rtcH264DepacketizationHandler->addToChain(rtcRTPDepacketizer);
                    // rtcH264DepacketizationHandler->addToChain(rtcRTCPReceivingSession);
                    rtcVideoTrack1->setMediaHandler(rtcH264DepacketizationHandler);

                    // Set the onMessage callback for the video track.
                    rtcVideoTrack1->onMessage(
                        [this](std::variant<rtc::binary, rtc::string> rtcMessage)
                        {
                            // Call the display frame for debugging.
                            if (std::holds_alternative<rtc::binary>(rtcMessage))
                            {
                                // Check the type of message.
                                if (rtcVideoTrack1->description().type() == "video")
                                {
                                    // Retrieve the binary message.
                                    rtc::binary rtcBinaryData = std::get<rtc::binary>(rtcMessage);
                                    LOG_INFO(logging::g_qSharedLogger, "Binary data size: {}", rtcBinaryData.size());
                                    // Print the binary data as a string.
                                    std::string szBinaryDataStr(reinterpret_cast<const char*>(rtcBinaryData.data()), rtcBinaryData.size());
                                    // std::cout << szBinaryDataStr << std::endl;
                                    LOG_INFO(logging::g_qSharedLogger, "Received binary data: {}", szBinaryDataStr);
                                }
                            }
                            LOG_WARNING(logging::g_qSharedLogger, "Received data from peer connection.");
                        });
                });

I don't know if this will help, but this is the data I get if I use no media handler:

\x90`]\xD10\xF5\x81<df\x0C\x91\xBE\xDE\x00\x03"\xCA]LA\x07\x16\x900\x00\x00\x00|\x01\x06\xF6\xB5\xFDc`\x11\xCE+O\x9F\xFF\xBE1a\x11$\xD2\xB6\xBFt\xE4}"b\x7F\xB7\xB5U*\xAB{\xE0;$q\xE4\x8C\xD3Pqhm\xF6/\xC0\xC1\x03\x956\x95o\x87\xFF}\x036\xD2\xFA\xF6\xB00:\xF5;\xF5?s\xC0\x06`\x7F\xB2\x9C\xFD\xFE\xD1Y\xF9\x9Ai\xDA\xDE\x05N\xFE)\xE9\x14\xF8\xF2o\xC9\xFAE.\x08\xD80A\xFF\xC3\xCAJ\xBC\x0F\xF7\xA3\xF1\x1D\x9A\x11Kj\x99\xFD\xE0?R\x97$S\xFF>\x0C\x0D\x8B\x9Bh\x7F\xC6{\xFE\x02$R&n\xCC\xBF\xAF\xD1<\x8A|?3\xFF\x9Ea(\xE3\xC3=\xF8@\x9F\xAE\xFE\x11\xC0!~\xBF\xF7\x00\x1F\xB7u\x9Fu\xEB\xC2oN_\xFF\xCF\x80\x18G\x95\xB8\xDD\xFF\xE0\x028\xED\xFB\xFF\xBC\x08\xB2\xC9C\xC5%7\xFF\xCA\x9A\o\xFC\x0C,\x8B\xD8v\xD2\xBEx\x04\xEB\xDB\xD2\xC0\xFC\x009\x00\xAD\xD4\x8B]\xFC\x7F\xEF\x82$q\x12\x0Ec\xD3\xFF\xD3\xEC\xA6\xC0\x01\x97s^\xBD\x83\x0C\x00^+\xA2\xBC\xA5I\xB4\xA6\x8D\x06a\x9F\x93\xD2k\x7F\x1B\x00Z\x9B$C\xF7\xEC\x11\xB7M\xB5I\xF7F\x9B\xBE3t\xF6\x04\xCE\xA1\xE0\xB4\xCA5\xF9O\xFE\xF7\x0Bb\x1C\xE9\xA8\x7F\xF8R\x06\xE7\x85\xB16\xA9\x11\xFF\xFF\xF7\xA7Z|\xE4\xEF$\x11\xC9W* \xFE,\x83\x90\xC6\xE44\'\xFB|\xBE\xE4\xD5\xFBo\xF7\xFA\xA0\x1A\x1C\x1B_'\xA4\x97\x88\xE3`\x0ER\x96\xDC\xD9\xDF\xD0a\xF0\x04\x12\x16\xBE\xF3{\xB4\x1D\xB9\xF6\x93\\xF7\xE0\x02\xFD\xDC\xAA\xB3\xFF\xFD\xC0<b\xCD\x1Cx\x9F\xF4\x0C?N :\xC2w~Z\xFA_\x1B\xC2\x0B8\xD3\xA5\xF7\x83\xC76\xE4\xB2\x1F\xFF\xF6\x15q\xAFZ\xEE9\xF6\x05\x14\xD9\x8F\xAA\x89^\xEC2K\xF7?\xFF\xE0\xC0\x0B\x93|\x931\xF1(1\xC7\x09r\xA0\x1B\xE1O\x86\x00\x19\x13W\xD7\xBB\x86!q]\xCD\x97\xAC\xFET\x8E\xCBS\xFE\xEB\x87\x03\x12\xF2@\x01=\xA5;\x9Fw\xFF\x06\xED&A\xA2\x01}``\x02\xB8\x92\xA7\xAF\xE2\x1F\xFB\xEA\xFF]\x81\xFE\x96\x1C\x0E\x81\x8Cv\x00\x13\xEF\xA7\xF4\xA5\xF40\xFE\x00\x1AIz\xAD\xEF\xEF\xC0\xE3\x12\xDE\x92\xAA=^\xFF[\xAF\x1B\x03JXYc\xDC\xDF\xFD\x82i\xE3x\xAB\xFF\xF7\xFBG'\xFD.\xF0>\xC1\x07\x98\xF3\x00\x1F9\xE2UJ\x9F\xAD\xAA\x11(Y$\x7F\xFB\xFD\x05\xB2\x9A22[\xF3\xFC:D\xBDj`\xEF;\xF8,Q\xD2c)#?\xDF\xD5\xFA\xBB]\x97\xAC0\xC0\x03\xD1\xA3T:\xFBe\xCA`|tP\x7F\xB5\xD4l\x01\xB4\xD5>w{\xC3\x00\x0CCMyi\xB7\x82(\xD7\xF5\xAE\xFB'\xBAu\xEF\xDF\xEA\x09-,\xBE:\x7F\xEF\xA0\x00\xBE^\xAF\xAE\xFF\x83\xDA\xDF-_~\x02\x1A\xD3\xA2=\x91\xEB\xE0\x1Di}\xFAN\xF80\xC0\x00\xBE\xE5\xD7\xC7\xCCrM\x1F}cw\x92\xE8\xD6\xA5\x83\x0C\x00-/\xFB\xAA\xDF\xBFx\x1A\xC4O\xB2\xBC\xF5'\xE8\x0C\xD4\xD7\xAD%\xEF\xFDcSr\x1E0\xB9\xFB\x80\x01\xFA\xE5\xEB\xED?n\xD9\xBE\xBF\xB9\\x1F\x86\x10Oki\xB9\x0B#\xF7\xE8\x8A:y\x8C\x958\x15/\x93\xE9\xF5\x04C0\xEC\x06?\xD3zu\xFC\x1E
?iU\xB7\xFB`\x07J\xE7\xEE\xEA\xC0\x12\xBEo4i\xE0\x8A\xDFk?\xDC\x94\xB2\xA9\xFD~\xC9\x96\x17\xF4\x1Ft\xDD\xCEN@k\x9C\xC1r_\xCAf\xFD\x06\xBC\xEF\xFC\x9E\x95/\xE3a\xB5\xA6\xE9>~T:\xB3\xF4:\xFB]q\xBF\xD86\xDFLvh\xFF\xF7\xD0\x00\xD1\xB9u\xF3\xFB\x95\xAF\x97\xB5\xCB\xE0\xC3\xFF\x00\x0B\xC3\xB2j\x92\x92W\xFE\xEF\x00\x0C\x8D\xCE\xD6\xD6\xBD]\xAE\x7F\xDC\x18.\xEF\xD3S\x7F\xFE\xF2/)/\x93\xD5)\x1E\xB1\xB0\xF0\x03 \xBBjk\xC0\x01p\xF7\xB2\xD9#\x9B\xBE\xD5/\x7F&\x9A\xB6,Hv\x9F\xEF\x1D\xEAu\x1D]\xD0\xE1\xA7\x04\x15[\xAEI\xE6\xFF\xDA\xC5\xDF\xB6\xA3\xA7\xFF\xEFd\xDB\xE9l|k\xB4g,\xAE\xDFw\xE1\x86c\xEBk\xB6\x07\x93L\xE9\xAC\x7F\xCF\xE45\x94 
8\x03\xF9>\x92}Tl\x14\xF92\xF9\xC7jF\xDE\xA91\x15,z\x11\x92wy{\xDD\xFA:\xEA\xA0\xE1\xF6\x7F\x09\x0E\xF4M\x97\xBB^\xF4~\xC3\xDD\x02&\xE8\x91\xF1\x88\xF6\xD8.j\xA3\xFF{\xADeW\xDA\xE8\xFDw\xD0\xC1\xD3\xEB-z~\xFF\xFEC\xCB\xB1\xFCE8_\x00\x02\xD3\xBE\xEA=\xFF\x02\x06\x8B\x97\xB7\x97\xA6\x96\x8E\xBF\xFE\x00zE\xDAEi\xBF\xEA\x12\xDE\xE4\xA6\xFE\xF4\xF00\xFF\x81m\x1F\x957\xE5\xB2\x94\x11\xE8\x07'\xDFM\xB7 \xDF\x80\x14\x05\x1B\xC7\x9F\x0F\xBF\xE7\xE0M<\xAF

This is the data I get if I use an RtpDepacketizer. It is consistently only 56 bytes, the first few of which are some sort of header I think:

\x80\xC8\x00\x06\x89\xA4\xEF\x8B\xEA\xE2\x7F\x92\x92\xA3&\xE1\xB5J@\x9B\x00\x00\x07q\x00\x1F\xAA\xB1\x81\xCA\x00\x06\x89\xA4\xEF\x8B\x01\x10kHZrjXLEXI/1srrK\x00\x00

This is the data I get if I use an H264RtpDepacketizer. It is consistently only 56 bytes, the first few of which are some sort of header I think:

\x80\xC8\x00\x06\xF9\x08\xC0/\xEA\xE2\x7F\xD5i\x97\xE5dqs1e\x00\x00\x08y\x00"\xE3e\x81\xCA\x00\x06\xF9\x08\xC0/\x01\x10sdyDEESmFcZcsUha\x00\x00

Here's the SDP offer from the unreal engine signalling server if you're curious:

v=0
o=- 4207938219345445375 2 IN IP4 127.0.0.1
s=-
t=0 0
a=group:BUNDLE 0 1 2
a=extmap-allow-mixed
a=msid-semantic: WMS pixelstreaming_audio_stream_id pixelstreaming_video_stream_id

m=video 9 UDP/TLS/RTP/SAVPF 96 97 98 99 100 101 102
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:8Dj7
a=ice-pwd:/M+UCXmwi0v8yumCI3dF1NGg
a=ice-options:trickle
a=fingerprint:sha-256 F8:DE:76:27:26:A6:A4:4E:88:EA:B3:CD:6F:E0:BA:0D:3F:23:08:03:C9:A3:E9:B3:7A:50:D5:A0:4D:5F:95:3D
a=setup:actpass
a=mid:0
a=extmap:1 urn:ietf:params:rtp-hdrext:toffset
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:3 urn:3gpp:video-orientation
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:5 http://www.webrtc.org/experiments/rtp-hdrext/playout-delay
a=extmap:6 http://www.webrtc.org/experiments/rtp-hdrext/video-content-type
a=extmap:7 http://www.webrtc.org/experiments/rtp-hdrext/video-timing
a=extmap:8 http://www.webrtc.org/experiments/rtp-hdrext/color-space
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=extmap:10 urn:ietf:params:rtp-hdrext:sdes:rtp-stream-id
a=extmap:11 urn:ietf:params:rtp-hdrext:sdes:repaired-rtp-stream-id
a=sendonly
a=msid:pixelstreaming_video_stream_id pixelstreaming_video_track_label
a=rtcp-mux
a=rtcp-rsize
a=rtpmap:96 H264/90000
a=rtcp-fb:96 goog-remb
a=rtcp-fb:96 transport-cc
a=rtcp-fb:96 ccm fir
a=rtcp-fb:96 nack
a=rtcp-fb:96 nack pli
a=fmtp:96 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42e01f
a=rtpmap:97 rtx/90000
a=fmtp:97 apt=96
a=rtpmap:98 H264/90000
a=rtcp-fb:98 goog-remb
a=rtcp-fb:98 transport-cc
a=rtcp-fb:98 ccm fir
a=rtcp-fb:98 nack
a=rtcp-fb:98 nack pli
a=fmtp:98 level-asymmetry-allowed=1;packetization-mode=1;profile-level-id=42001f
a=rtpmap:99 rtx/90000
a=fmtp:99 apt=98
a=rtpmap:100 red/90000
a=rtpmap:101 rtx/90000
a=fmtp:101 apt=100
a=rtpmap:102 ulpfec/90000
a=ssrc-group:FID 402974310 2167710514
a=ssrc:402974310 cname:bRUcICksSsdBkpb3
a=ssrc:402974310 msid:pixelstreaming_video_stream_id pixelstreaming_video_track_label
a=ssrc:2167710514 cname:bRUcICksSsdBkpb3
a=ssrc:2167710514 msid:pixelstreaming_video_stream_id pixelstreaming_video_track_label

m=audio 9 UDP/TLS/RTP/SAVPF 111 63 110
c=IN IP4 0.0.0.0
a=rtcp:9 IN IP4 0.0.0.0
a=ice-ufrag:8Dj7
a=ice-pwd:/M+UCXmwi0v8yumCI3dF1NGg
a=ice-options:trickle
a=fingerprint:sha-256 F8:DE:76:27:26:A6:A4:4E:88:EA:B3:CD:6F:E0:BA:0D:3F:23:08:03:C9:A3:E9:B3:7A:50:D5:A0:4D:5F:95:3D
a=setup:actpass
a=mid:1
a=extmap:14 urn:ietf:params:rtp-hdrext:ssrc-audio-level
a=extmap:2 http://www.webrtc.org/experiments/rtp-hdrext/abs-send-time
a=extmap:4 http://www.ietf.org/id/draft-holmer-rmcat-transport-wide-cc-extensions-01
a=extmap:9 urn:ietf:params:rtp-hdrext:sdes:mid
a=sendrecv
a=msid:pixelstreaming_audio_stream_id pixelstreaming_audio_track_label
a=rtcp-mux
a=rtpmap:111 opus/48000/2
a=rtcp-fb:111 transport-cc
a=fmtp:111 maxaveragebitrate=510000;maxplaybackrate=48000;minptime=3;sprop-stereo=1;stereo=1;usedtx=0;useinbandfec=1
a=rtpmap:63 red/48000/2
a=fmtp:63 111/111
a=rtpmap:110 telephone-event/48000
a=maxptime:120
a=ptime:20
a=ssrc:3808006951 cname:bRUcICksSsdBkpb3
a=ssrc:3808006951 msid:pixelstreaming_audio_stream_id pixelstreaming_audio_track_label

m=application 9 UDP/DTLS/SCTP webrtc-datachannel
c=IN IP4 0.0.0.0
a=ice-ufrag:8Dj7
a=ice-pwd:/M+UCXmwi0v8yumCI3dF1NGg
a=ice-options:trickle
a=fingerprint:sha-256 F8:DE:76:27:26:A6:A4:4E:88:EA:B3:CD:6F:E0:BA:0D:3F:23:08:03:C9:A3:E9:B3:7A:50:D5:A0:4D:5F:95:3D
a=setup:actpass
a=mid:2
a=sctp-port:5000
a=max-message-size:262144

Link to my code: https://github.com/MissouriMRDT/Autonomy_Software/blob/topic/sim-cam/src/vision/cameras/sim/SIMZEDCam.cpp

[paullouisageneau]

You need to use H264RtpDepacketizer. When setting a depacketizer, frames are not delivered by onMessage but by onFrame, the reason is that you need to also receive metadata like timestamps. What you get with onMessage when using a depacketizer are RTCP messages, which are not intercepted by a media handler.

[ClayJay3]

Ah, okay, that makes sense. I switched to using onFrame(), and I seem to be getting something that looks more like reconstructed H264 frame data now:

2024-11-17 10:44:51.626 INFO      [191187] [SIMZEDCam.cpp:439] FrameInfo timestamp: 4250317607, payload type: 96
2024-11-17 10:44:51.626 INFO      [191187] [SIMZEDCam.cpp:441] Received binary message of size: 22322

New OnTrack() callback

m_pPeerConnection->onTrack(
                [this](std::shared_ptr<rtc::Track> rtcTrack)
                {
                    // Submit logger message.
                    rtc::Description::Media rtcMediaDescription = rtcTrack->description();
                    // Get some information about the track.
                    std::string szMediaType = rtcMediaDescription.type();

                    // Check if the track is a video track.
                    if (szMediaType != "video")
                    {
                        return;
                    }

                    // Set member variable to the video track.
                    rtcVideoTrack1 = rtcTrack;

                    // Create a H264 depacketization handler and rtcp receiving session.
                    rtcTrack1H264DepacketizationHandler = std::make_shared<rtc::H264RtpDepacketizer>();
                    // rtcRTCPReceivingSession             = std::make_shared<rtc::RtcpReceivingSession>();
                    // rtcTrack1H264DepacketizationHandler->addToChain(rtcRTCPReceivingSession);
                    rtcVideoTrack1->setMediaHandler(rtcTrack1H264DepacketizationHandler);

                    // Set the onMessage callback for the video track.
                    rtcVideoTrack1->onFrame(
                        [this](rtc::binary rtcBinaryMessage, rtc::FrameInfo rtcFrameInfo)
                        {
                            // Print frame info timestamp and payload type.
                            LOG_INFO(logging::g_qSharedLogger, "FrameInfo timestamp: {}, payload type: {}", rtcFrameInfo.timestamp, rtcFrameInfo.payloadType);
                            // Print the size of the binary message.
                            LOG_INFO(logging::g_qSharedLogger, "Received binary message of size: {}", rtcBinaryMessage.size());
                            // Change the rtc::Binary (std::vector<std::byte>) to a std::vector<uint8_t>.
                            std::vector<uint8_t> vH264EncodedBytes;
                            vH264EncodedBytes.reserve(rtcBinaryMessage.size());
                            for (std::byte stdByte : rtcBinaryMessage)
                            {
                                vH264EncodedBytes.push_back(static_cast<uint8_t>(stdByte));
                            }
                            // Acquire lock on the WebRTC copy mutex.
                            std::unique_lock<std::shared_mutex> lkWebRTC(m_muWebRTCCopyMutex);
                            // Copy the H264 image to the cv::Mat. Image dims are 1280x720 and 4 channels.
                            // m_cvFrame = cv::Mat(720, 1280, CV_8UC4, vH264EncodedBytes.data());

                            // Decode the H264 encoded bytes to a cv::Mat.
                            this->DecodeH264BytesToCVMat(vH264EncodedBytes, m_cvFrame);
                        });
                });

When I feed this binary data to an FFMPEG for decoding into something usable for OpenCV, I get an error from the decoder about SPS PPS missing for the frame data.

2024-11-17 10:58:59.549 INFO      [194997] [SIMZEDCam.cpp:439] FrameInfo timestamp: 983365100, payload type: 96
2024-11-17 10:58:59.549 INFO      [194997] [SIMZEDCam.cpp:441] Received binary message of size: 26137
[h264 @ 0x7415d40efcc0] non-existing PPS 0 referenced
[h264 @ 0x7415d40efcc0] decode_slice_header error
[h264 @ 0x7415d40efcc0] no frame!
Failed to send packet to decoder!
2024-11-17 10:58:59.562 INFO      [194995] [SIMZEDCam.cpp:439] FrameInfo timestamp: 983366810, payload type: 96
2024-11-17 10:58:59.562 INFO      [194995] [SIMZEDCam.cpp:441] Received binary message of size: 20649
[h264 @ 0x7415d80ef3c0] non-existing PPS 0 referenced
[h264 @ 0x7415d80ef3c0] decode_slice_header error
[h264 @ 0x7415d80ef3c0] no frame!
Failed to send packet to decoder!

Does the H264RtpDepacketizer strip the SPS/PPS data off? Is there somewhere I can find it? I tried to append 0x00 0x00 0x00 0x01 0x00 0x00 0x00 0x01 without much luck.

To try and debug it, I wrote one of the receiving frames to an ouput .bin file and then used ffmpeg -i raw_frame.bin -f null -. Here's the output:

root@PowerStation:/workspaces/Autonomy_Software/build# ffmpeg -i raw_frame.bin -f null -
ffmpeg version n7.1 Copyright (c) 2000-2024 the FFmpeg developers
  built with gcc 10 (Ubuntu 10.5.0-1ubuntu1~22.04)
  configuration: --prefix=/tmp/pkg/ffmpeg_7.1_amd64/usr/local --enable-static --disable-shared --disable-doc --enable-gpl --enable-libx264 --enable-pic
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.100 / 61. 19.100
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
[h264 @ 0x6498e6df7f80] Invalid level prefix
[h264 @ 0x6498e6df7f80] error while decoding MB 79 3
[h264 @ 0x6498e6df7f80] concealing 3330 DC, 3330 AC, 3330 MV errors in I frame
Input #0, h264, from 'raw_frame.bin':
  Duration: N/A, bitrate: N/A
  Stream #0:0: Video: h264 (Constrained Baseline), yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], 25 fps, 120 tbr, 1200k tbn
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> wrapped_avframe (native))
Press [q] to stop, [?] for help
[h264 @ 0x6498e6e10d00] Invalid level prefix
[h264 @ 0x6498e6e10d00] error while decoding MB 79 3
[h264 @ 0x6498e6e10d00] concealing 3330 DC, 3330 AC, 3330 MV errors in I frame
[vist#0:0/h264 @ 0x6498e6dfa100] [dec:h264 @ 0x6498e6e2d080] corrupt decoded frame
Output #0, null, to 'pipe:':
  Metadata:
    encoder         : Lavf61.7.100
  Stream #0:0: Video: wrapped_avframe, yuv420p(progressive), 1280x720 [SAR 1:1 DAR 16:9], q=2-31, 200 kb/s, 60 fps, 60 tbn
      Metadata:
        encoder         : Lavc61.19.100 wrapped_avframe
[out#0/null @ 0x6498e6dfb380] video:0KiB audio:0KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: unknown
frame=    1 fps=0.0 q=-0.0 Lsize=N/A time=00:00:00.01 bitrate=N/A speed=3.68x

[ClayJay3]

I printed out the raw byte data to my console and noticed that the frame does actually have a NAL start unit, just no SPS or PPS units, which are needed by the ffmpeg decoder.

0 0 0 1 6 1 4 0 b6 8 10 80 0 0 0 1 61 eb 60 9f 44 96 8 1a 1d 61 fb 3b 68 11 4 1 18 4c 64 71 9 3a 4f fe 43 28 54 58

I did some more debugging and found that the first frame received seems to have the SPS (0x00 0x00 0x00 0x01 0x67) and PPS (0x00 0x00 0x00 0x01 0x68) data, and an IDR? (0x00 0x00 0x00 0x01 0x65)

00 00 00 01 67 42 C0 20 95 90 05 00 5B B0 11 00 00 03 03 E8 00 01 D4 C0 E0 00 00 09 89 68 00 00 26 25 A1 BB CB 83 B4 11 08 C9 00 00 00 00 01 68 CB 8F 20 00 00 00 01 06 00 07 81 5F 90 00 00 03 00 40 80 00 00 00 01 06 01 04 00 00 08 10 80 00 00 00 01 65 B8 04 05 7C 21 87 40 00 71 44 1C 03 F4 C6 FB FF 84 30 F5 99 ............

The second frame then starts off with an IDR (0x00 0x00 0x00 0x01 0x65) followed by a bunch of data.

And then, after the second frame, the rest of the frames always start with:

00 00 00 01 06 01 04 00 1C 08 10 80 00 00 00 01 61 ........

To me, this looks like a good H264 stream from libdatachannel's H264RtpDepacketizer, but FFMPEG doesn't seem to want to decode it. Any advice?

[paullouisageneau]

The h264 frames look as expected. Depending of how the sender is implemented, it is possible that the first frame is not an IDR, in that case the decoder has to wait for an IDR to start decoding. However, if the decoder can't start decoding after the first IDR frame, something is probably wrong with the ffmpeg decoding code. In any case, you should not tinker with the frames.

After a quick look to your code, it looks like the whole logic in the onFrame callback is wrong and unnecessary. You must not try to parse NAL units, strip separators, or insert SPS/PPS. You should simply feed rtcBinaryMessage.data() to the decoder.

[ClayJay3]

The H264 depacketizer works, it was just an issue with sending the data to the ffmpeg decoder.