Make multiple videos play in sync

[pictriage]

I have a project where there are multiple videos of the same event (from different angles), and I need to play them all synchronized. Since the videos start at different times, I have used audio analysis to identify how much each video needs to be offset by in order to play them in sync. For example:

Video 1: 0 seconds
Video 2: 3.35 seconds
Video 3: 7.23 seconds
Video 4: 9.10 seconds

Basically what I'm trying to do is automatically launch 4 MPV windows, and seek each video to the appropriate timestamp, then play all videos in sync. (And repeat this for many sets of videos on the fly.) I have a good GPU/CPU and have gotten this working with VLC, but with MPV I haven't had as much success yet. (I cannot use VLC for other reasons and I find MPV much smoother to work with overall.) With some small videos it works, but usually videos play out of sync. Usually there is at least 0.05s discrepancy, but up to 0.5 seconds or more. I'm hoping to keep it under 0.05 seconds.

Here is a MWE of the issue. Run this script, and then while the videos are playing, take a screenshot. I have an overlay of time stats on each video. Paste the screenshot somewhere, and you will see the timestamps are out of sync (not staggered by 10 seconds as they should be).

You can run this script by placing any video (preferably high bitrate like 4k or 1080p) in your working dir (it is named 'video.mp4' in the below script. Here is an example file. As far as I can see, the issue is more pronounced when you play more videos together, and when the videos have higher bitrates. Again, i am able to get the sync working pretty much perfectly with VLC, so I think my system is fundamentally capable of it. Wondering if anyone can improve on the below script? thank you so much.

import mpv
from dataclasses import dataclass
from time import sleep

@dataclass
class Video:
    name: str
    time_offset: float
    player: mpv.MPV = None


videos = [    
    Video(name='video.mp4', time_offset=10),
    Video(name='video.mp4', time_offset=20),
    Video(name='video.mp4', time_offset=30),
    Video(name='video.mp4', time_offset=40),
]

pixel = 0

VIDEO_HEIGHT = 540

for v in videos:
    player = mpv.MPV(
        ontop=True,
        # --hr-seek
        geometry=f'{VIDEO_HEIGHT}x{VIDEO_HEIGHT}+0+{pixel}',
        hr_seek='yes',
        osd_level=2,
        osd_fractions=True,
        #osd_font_size=12
        #pause=True,
    )
    player.play(v.name)
    m_offset, s_offset = divmod(v.time_offset, 60)
    player.osd_overlay(
        1, f'.\n\n+{int(m_offset)}:{s_offset:05.2f}')

    v.player = player
    pixel += 200

    while player.core_idle:
        sleep(0.05)


for v in videos:
    player = v.player
    player.seek(v.time_offset, reference='absolute', precision='exact')

    player.paused = True

for v in videos:
    v.player.paused = False

import time

while True:
    time.sleep(10)

[kasper93]

I never tried to synchronize players. If you need frame perfect synchronization, there might be an issues with slight playback speed variation between players, but most importantly dropped frames.

Although with audio sync we really should be keeping playback speed the same and probably would be good to find root cause why it desync for you exactly.

As a solution you probably can observe time-pos and calculate the error (delay) of each video to reference one and adjust the speed on each update to let the other videos catch up.

I would also try to play it in one player if it is suitable for your use-case for example 3 videos stacked --lavfi-complex='[vid1] [vid2] [vid3] hstack=inputs=3 [vo]'.

There is also https://github.com/Syncplay/syncplay but I think their design goals are not to synchronize frame perfect either, so probably not suitable for your use.

[po5]

My groupwatch_sync script allows you to automatically unpause the player at a given time, and get back in sync if you ever desync due to pausing, seeking or whatever else.
Schedule a start time with unix timestamp: script-message-to groupwatch_sync start-time 1700075500
Jump to the current group position: script-binding groupwatch_jump
There are more features but I doubt they'd be useful for scripting, only user-facing stuff.

[pictriage]

Hi, thanks so much for the replies. @kasper93 I got some improvement from doing as you suggested by monitoring time_pos and adjusting repeatedly, until the accuracy is within a reasonable limit. It doesn't always resolve the de-sync but is better than it was before. I'm going to continue looking for a more effective technique.

    for x in range(30):
        sleep(0.2)
        deltas = []
        for cam in cams:
            d = cam.player.time_pos - cam.time_offset
            deltas.append(d)
        print('adjustments:')

        for i, cam in enumerate(cams):
            # sync to the last one, because that's the one we measure last,
            # so it will be furthest along.
            adjustment = deltas[-1] - deltas[i]
            print(adjustment)
            if abs(adjustment) > 0.05:
                cam.player.time_pos += adjustment

@po5 thanks! I will check out your script.

[pictriage]

I am still experiencing a lot of out-of-sync video behavior (often by half a second or more). I just wonder what is the main reason for the videos to get out of sync? That would help me understand how to improve on what I have now. My script pauses the videos and uses exact-precision seeking to sync them. At that point, my screenshots confirm that the videos are almost exactly in sync (<0.01s).

However, after this line, the video get way out of sync:

for v in videos:
    v.player.paused = False

This loop takes 0.05-0.15 seconds to execute sequentially. That's not enough to explain the large de-sync of 0.5s or more (and if I use threading to un-pause all videos in parallel, it doesn't fix the issue).

So I guess un-pausing takes a variable amount of time, and some videos seem to take consistently longer to un-pause than others. Maybe related to whether the playback is near a keyframe or not?

(Again I have a proof-of-concept of this working with VLC, so I don't think the limitation is with my hardware or OS scheduler.)

[pictriage]

I've found a technique that seems to help. First, play the videos for 1 second. then measure each player's time_pos. Then go back and re-seek to the same spot, and start each video with an individualized delay to correct for how much it differed from the others on the first pass. Still not perfect but definitely an improvement.

    for cam in cams:
        cam.player.seek(max(0, cam.trim_offset), reference='absolute', precision='exact')

    threads = []
    for cam in cams:
        t = threading.Thread(target=cam.play_1_second)
        t.start()
        threads.append(t)
        
    for t in threads:
        t.join()

    head_starts = []
    for cam in cams:
        head_start = (cam.player.time_pos - cam.trim_offset)
        head_starts.append(head_start)

    for i, cam in enumerate(cams):
        cam.head_start = head_starts[i] - min(head_starts)

        cam.player.seek(max(0, cam.trim_offset), reference='absolute', precision='exact')

    for cam in cams:
        t = threading.Thread(target=cam.play_delayed)
        t.start()

[LetTheHavenCry]

I have a similar issue, I try to get two player to sync as close as possible, but still not prefect. I end up with using different thread for players and tell them to start at a position at the next x ticks or so, if I just use a single thread to loop for start the players, a lot of time will end up with higher desync, using thread will be a lot better.