Videos recorded with openvidu v2.27.0 and later are corrupted and cause playback issues to chromium based browsers

[kazepis]

Describe the bug
Videos recorded via openvidu recording API starting from openvidu v2.27.0 are corrupted and cause problems to chromium based browsers.

Expected behavior
When validating a recorded video using the mkvalidator tool, the tool should not report a problem. e.g. with openvidu v2.26.0 and older:

C:\OpenViduRecordings\7cc3d50a-9830-4734-81cf-3236b2cb4ea2>mkvalidator str_CAM_EL2h_con_G0nW89003P.webm
mkvalidator 0.6.0: the file appears to be valid
        file "str_CAM_EL2h_con_G0nW89003P.webm"
        created with Lavf57.83.100 / Lavf57.83.100

C:\OpenViduRecordings\7cc3d50a-9830-4734-81cf-3236b2cb4ea2>

Wrong current behavior
When you check a recorded video with the mkvalidator tool it reports many errors. e.g.: with openvidu version v2.27.0 - v2.29.0 (include major kurento update to 7.0.1)

C:\OpenViduRecordings\53a5fc2c-5bbb-4153-8962-2370b0889f99>mkvalidator str_CAM_A5Fw_con_AZ9WQ8qlCi.webm
ERR201: Invalid 'Colour' for profile 'webm' in Video at 348
ERR201: Invalid 'Range' for profile 'webm' in Colour at 365
ERR201: Invalid 'MatrixCoefficients' for profile 'webm' in Colour at 365
ERR201: Invalid 'TransferCharacteristics' for profile 'webm' in Colour at 365
ERR201: Invalid 'Primaries' for profile 'webm' in Colour at 365
ERR0B1: SimpleBlock at 2845 track #2 is not a keyframe
...
...
truncated
...
...
ERR0B1: SimpleBlock at 1889110 track #2 is not a keyframe
ERR312: CueEntry Track #1 and timecode 1018 ms not found
        file "str_CAM_A5Fw_con_AZ9WQ8qlCi.webm"
        created with GStreamer matroskamux version 1.16.3 / GStreamer Matroska muxer

C:\OpenViduRecordings\53a5fc2c-5bbb-4153-8962-2370b0889f99>

OpenVidu tutorial where to replicate the error
Use default openvidu recording tutorial for node: https://docs.openvidu.io/en/stable/tutorials/openvidu-recording-node/ to record a video and then inspect the video using the mkvalidator tool:

C:\OpenViduRecordings\ses_M4LFmfmu7k>mkvalidator --version
mkvalidator v0.6.0, Copyright (c) 2010-2020 Matroska Foundation
        file "--version"

C:\OpenViduRecordings\ses_M4LFmfmu7k>mkvalidator str_CAM_TAlt_con_Gf6TRaylW0.webm
ERR201: Invalid 'Colour' for profile 'webm' in Video at 348
ERR201: Invalid 'Range' for profile 'webm' in Colour at 365
ERR201: Invalid 'MatrixCoefficients' for profile 'webm' in Colour at 365
ERR201: Invalid 'TransferCharacteristics' for profile 'webm' in Colour at 365
ERR201: Invalid 'Primaries' for profile 'webm' in Colour at 365
ERR0B1: SimpleBlock at 517 track #2 is not a keyframe
ERR0B1: SimpleBlock at 4416 track #2 is not a keyframe
ERR0B1: SimpleBlock at 4483 track #2 is not a keyframe
ERR0B1: SimpleBlock at 11511 track #2 is not a keyframe
....
...
truncated
...
...
ERR0B1: SimpleBlock at 868702 track #2 is not a keyframe
ERR0B1: SimpleBlock at 868776 track #2 is not a keyframe
ERR312: CueEntry Track #1 and timecode 10 ms not found
        file "str_CAM_TAlt_con_Gf6TRaylW0.webm"
        created with GStreamer matroskamux version 1.16.3 / GStreamer Matroska muxer

C:\OpenViduRecordings\ses_M4LFmfmu7k>

OpenVidu deployment info
Docker container as explained in tutorial https://docs.openvidu.io/en/stable/tutorials/openvidu-recording-node/

docker run -p 4443:4443 --rm \
    -e OPENVIDU_SECRET=MY_SECRET \
    -e OPENVIDU_RECORDING=true \
    -e OPENVIDU_RECORDING_PATH=/opt/openvidu/recordings \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v C:/OpenViduRecordings:/opt/openvidu/recordings \
openvidu/openvidu-dev:2.29.0

IMPORTANT NOTE: please, if you think the bug might be related to OpenVidu Server side, specify here if you are also getting the error by using OpenVidu Server Demos instance. This instance is publicly available (use it only for this test, because it is not secure!!!): URL: https://demos.openvidu.io:4443, SECRET: MY_SECRET

Cannot test with this one because i cannot get the recording file

Client device info (if applicable)
Describe the client device(s) or platform(s) where you are able to replicate the error. For example:

• Chrome Version 118.0.5993.118 (Official Build) (64-bit) on Windows 10 Pro (22H2).

Additional context
The above corrupted video seems to be affecting chromium based browser because when I try to play it back using this html code:

<!DOCTYPE html> 
<html> 
<head>

  </head>
<body> 

<video width="320" height="240" controls>
  <source src="str_CAM_TAlt_con_Gf6TRaylW0.webm" type="video/webm">
  Your browser does not support the video tag.
</video>

<script>
const allEvents = [
'abort',
'canplay',
'canplaythrough',
'durationchange',
'emptied',
'ended',
'loadeddata',
'loadedmetadata',
'loadstart',
'pause',
'play',
'playing',
'progress',
'ratechange',
'resize',
'seeked',
'seeking',
'stalled',
'suspend',
'timeupdate',
'volumechange',
'waiting'
];
const video = document.querySelector("video");

allEvents.forEach((e) => {
  video.addEventListener(e, (event) => {
    console.log(e + " event fired, ready state: ", video.readyState);
  });
});

video.addEventListener("error", e => {
  console.error(e);
  console.error(e.target);
  console.error(e.target.error);
});

//video.src = "str_CAM_TAlt_con_Gf6TRaylW0.webm";
//video.load();

</script>
</body>

</html>

I get this error:

After the error, any attempt to seek to a timestamp does not trigger the canplaythrough event ANY MORE.

I remain at your disposal for any more information you may need.

[kazepis]

A difference that was noticed was the reported muxer:

Openvidu v2.26.0 / Kurento 6

created with GStreamer matroskamux version 1.8.1.1 / GStreamer Matroska muxer

Openvidu v2.27.0 - v2.29.0 / Kurento 7.0.1

created with GStreamer matroskamux version 1.16.3 / GStreamer Matroska muxer

[kazepis]

Good morning everybody! Has anyone seen this and if so were you able to reproduce?

[cruizba]

Hello @kazepis,
You previously stated that your deployment is this:

docker run -p 4443:4443 --rm \
    -e OPENVIDU_SECRET=MY_SECRET \
    -e OPENVIDU_RECORDING=true \
    -e OPENVIDU_RECORDING_PATH=/opt/openvidu/recordings \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v C:/OpenViduRecordings:/opt/openvidu/recordings \
openvidu/openvidu-dev:2.29.0

Is this the only way of deployment? Or do you have a production environment?

If you have a production environment, which edition is it? CE, PRO, or ENTERPRISE.

@j1elo has a suggestion which is to change a Kurento configuration value, specifically gapsFix=GENPTS. But to give you instructions on how to configure Kurento, we need to know what types of deployment you have, as the way to change the configuration may vary.

[j1elo]

Hi, I was looking at this today.

Seeing this output from mkvalidator:

ERR201: Invalid 'Colour' for profile 'webm' in Video at 348
ERR201: Invalid 'Range' for profile 'webm' in Colour at 365
ERR201: Invalid 'MatrixCoefficients' for profile 'webm' in Colour at 365
ERR201: Invalid 'TransferCharacteristics' for profile 'webm' in Colour at 365
ERR201: Invalid 'Primaries' for profile 'webm' in Colour at 365

and guided by your suggestion to look into the GStreamer matroskamux component, I've found an awfully similar bug report in the issue tracker of GStreamer (i.e. the 3rd-party library that actually handles media within Kurento):

https://gitlab.freedesktop.org/gstreamer/gstreamer/-/issues/2526

I cannot do much right now regarding this, but at least I confirmed the issue so GStreamer devs see that it affects more people. As a workaround, I'm thinking that Kurento has a couple ways to deal with packet loss wrt. recording; the default is to leave missing pieces of video/audio as gaps in the file, but another option is to enable the "GENPTS" GapsFix method, which attempts to fix timestamps and provide a smooth playback in web browsers.

This recording mode is enabled by editing a Kurento config file: RecorderEndpoint.conf.ini. It's however not been exposed as part of the OpenVidu API, so depending on your deployment, editing such file can be more or less cumbersome.

There is a Kurento guide to deal with config file modifications in Docker environments: https://doc-kurento.readthedocs.io/en/latest/user/faq.html#faq-docker

but probably it's better to follow the more specific help that @cruizba can offer in this regard.

[kazepis]

Thank you both @cruizba and @j1elo.
I was able to repro the problem both when on my dev laptop on windows 10 (with docker desktop and the docker run command that @cruizba is quoting) AND when trying with the on-premise "production" env which has a CE version deployed on a VM with Ubuntu Server LTS 22.04.3. I tried all versions of Openvidu from 2.27 to 2.29 and they all seem to have the same problem with the recorded videos.

I am more than confident that I will be able to perform any trials if you give me sufficient instructions. I will be able to attach to a running docker container and change configuration. Just tell me how to "reload" the changed configuration.

Thank you very much for your time and effort and I remain at your disposal.

[kazepis]

@cruizba and @j1elo I tried what you suggested but the problem remains the same.
I edited the docker_compose.yml to expose the kurento config to local disc by adding the following line in the volume section of kms container:

 - /opt/openvidu/kurento-config:/etc/kurento/modules/kurento

Then I copied all the contents of /etc/kurento/modules/kurento to local disc using the command docker cp openvidu-kms-1:/etc/kurento/modules/kurento/ /opt/openvidu/kurento-config/

I edited the RecorderEndpoint.conf.ini to perform the change you proposed:

root@openvidu2:/opt/openvidu/kurento-config# cat RecorderEndpoint.conf.ini
;; How to fix gaps when they are found in the recorded stream.
;;
;; Gaps are typically caused by packet loss in the input streams, such as when
;; an RTP or WebRTC media flow suffers from network congestion and some packets
;; don't arrive at the media server.
;;
;; Possible values:
;;
;; * NONE: Do not fix gaps.
;;
;;   Leave the stream as-is, and store it with any gaps that the stream might
;;   have. Some players are clever enough to adapt to this during playback, so
;;   that the gaps are reduced to a minimum and no problems are perceived by the
;;   user; other players are not so sophisticated, and will struggle trying to
;;   decode a file that contains gaps. For example, trying to play such a file
;;   directly with Chrome will cause lipsync issues (audio and video will fall
;;   out of sync).
;;
;;   This is the best choice if you need consistent durations across multiple
;;   simultaneous recordings, or if you are anyway going to post-process the
;;   recordings (e.g. with an extra FFmpeg step).
;;
;; * GENPTS: Adjust timestamps to generate a smooth progression over all frames.
;;
;;   This technique rewrites the timestamp of all frames, so that gaps are
;;   suppressed. It provides the best playback experience for recordings that
;;   need to be played as-is (i.e. they won't be post-processed). However,
;;   fixing timestamps might cause a change in the total duration of a file. So
;;   different recordings from the same session might end up with slightly
;;   different durations.
;;
;; Default: NONE.
;;
gapsFix=GENPTS
root@openvidu2:/opt/openvidu/kurento-config#

I restarted openvidu with ./openvidu restart and recorded a new video call.

However, the playback is still halted on Chrome and the matroska validator still complains:

C:\OpenViduRecordings>mkvalidator --no-warn str_CAM_VQtf_con_IPivS72avr.webm
ERR201: Invalid 'Colour' for profile 'webm' in Video at 348
ERR201: Invalid 'Range' for profile 'webm' in Colour at 365
ERR201: Invalid 'MatrixCoefficients' for profile 'webm' in Colour at 365
ERR201: Invalid 'TransferCharacteristics' for profile 'webm' in Colour at 365
ERR201: Invalid 'Primaries' for profile 'webm' in Colour at 365
ERR0B1: SimpleBlock at 4360 track #2 is not a keyframe
ERR0B1: SimpleBlock at 5510 track #2 is not a keyframe
ERR0B1: SimpleBlock at 5575 track #2 is not a keyframe
...
...
truncated
...
...
ERR0B1: SimpleBlock at 11504170 track #2 is not a keyframe
ERR0B1: SimpleBlock at 11508164 track #2 is not a keyframe
        file "str_CAM_VQtf_con_IPivS72avr.webm"
        created with GStreamer matroskamux version 1.16.3 / GStreamer Matroska muxer

C:\OpenViduRecordings>

Is there any way I can try to use the latest gstreamer in the kurento container to see if the problem is fixed?