Timewarp Producer Issue: Ignores framerate settings in mlt_profile, reports wrong frame on callback.

[Eoin-ONeill-Yokai]

I've been trying to use a timewarp producer to allow for fast-forward and slow-motion features in my current application and noticed some issues after hot swapping my existing producer (avformat) with a timewarp producer wrapping my original producer.

Firstly, I noticed that my callback consumer-frame-show no longer respects my current frame rate settings (i.e. when I change my profile to 4 frames per second, I still get 24 frame show callbacks a second that I have to deal with) which is causing some parts of my application to not respect the target framerate settings. I've even tried to reassign the profile after changes to frame-rate, but it would still always callback 24 times a second.

Secondly, I want to know if there's a good way to discern the position of the frame relative to the internal producer (the avformat producer) from within the consumer-frame-show callback. I've noticed that the timewarp obviously changes the frame's "position" to report the position of the timewarp producer. While I could perhaps discern the position using a simple multiplication (timewarp warp_speed value * frame position) I can't help but wonder if there's a better way to do this.

[ddennedy]

when I change my profile to 4 frames per second, I still get 24 frame show callback

Really? That seems impossible. The consumer fires this event according to its framerate, which can be selected by using a profile. It does not know about the timewarp producer. It sounds like your profile was adversely affected. (Not sure why you want 4 fps?!?) I do not believe we see this problem in Shotcut. Did you try using Shotcut?

[bmatherly]

when I change my profile to 4 frames per second, I still get 24 frame show callbacks a second that I have to deal with

Are you using that new profile to initialize all of your services? Or just the producer?

I want to know if there's a good way to discern the position of the frame relative to the internal producer

You should be able to query the "original_position" property on the frame:

mlt/src/framework/mlt_frame.c

Line 183 in 9c3bb7d

if ( ! mlt_properties_get( MLT_FRAME_PROPERTIES( self ), "original_position" ) )

[ddennedy]

Also, if you change the profile after you start the consumer, it will not respond to those changes. You need to restart the consumer.

[ddennedy]

You should be able to query the "original_position" property on the frame:

See also mlt_frame_original_position()

[Eoin-ONeill-Yokai]

@ddennedy

It sounds like your profile was adversely affected. (Not sure why you want 4 fps?!?) I do not believe we see this problem in Shotcut.

I'm working on Krita and we're currently in the process of porting its animation system from QtMultimedia to MLT. Krita animators typically work at around 24fps but they also want to be able to preview their animation at any arbitrary frame rate.

Also, if you change the profile after you start the consumer, it will not respond to those changes. You need to restart the consumer.

@bmatherly

Are you using that new profile to initialize all of your services? Or just the producer?

I'm positive that the profile for both the producer and the consumer have the correct frame rate values. I'm also stopping and resuming any time the framerate value is changed by the user. Again, when not using timewarp, the behaviour is as expected. Here's an output log from within our callback where I'm getting the elapsed time in MS compared to the profile settings of the consumer and producer.

Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41
Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41
Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41
Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41
Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41
Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41
Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41
Entering "mlt_frame)()" consumer.producer()->profile()->frame_rate_num() = 4 consumer.profile()->frame_rate_num() = 4 self->temp_stopWatch() = 41

As you can see, the elapsed time between each callback is 41ms apart (1/24th of 1000ms) while the consumer and producer both seem to correctly report the right frame rate numerator (denominator is always 1 in our current use case.) Here's a snapshot of our frame rate assignment method.

void KisMediaConsumer::setFrameRate(int fps)
{
    StopAndResumeConsumer srPullConsumer(m_d->pullConsumer.data());
    StopAndResumeConsumer srPushConsumer(m_d->pushConsumer.data());

    m_d->profile->set_frame_rate(fps, 1);
}

The class StopAndResumerContainer is a utility class that I've made to use scopes to stop a running container and automatically resume it when going out of scope. But essentially, I am absolutely sure that consumers are being stopped and restarted after each change.

So I'm not entirely sure what I've done wrong with the current implementation. This is all for a WIP merge request, by the way, so all of the MLT related code is currently here:
KisMediaConsumer.cpp

Our setup is slightly unconventional because we want to preserve the way we're currently interacting with Krita's built-in canvas / image api.

[bmatherly]

I'm working on Krita and we're currently in the process of porting its animation system from QtMultimedia to MLT.

That is exciting!

void KisMediaConsumer::setFrameRate(int fps)
{
    StopAndResumeConsumer srPullConsumer(m_d->pullConsumer.data());
    StopAndResumeConsumer srPushConsumer(m_d->pushConsumer.data());

    m_d->profile->set_frame_rate(fps, 1);
}

It is not sufficient to stop and restart the consumer to change the frame rate. You actually have to destroy and recreate all the services (producers, filters, consumers). The reason is that a service may use the profile in its constructor and then never adjust to changes.

For example, the timewarp producer uses the profile frame rate on construction and never updates if the profile changes:
https://github.com/mltframework/mlt/blob/master/src/modules/core/producer_timewarp.c#L249

I suspect something similar is happening with your consumer - which is why the 24Hz callback persists.
It looks like the consumer has some code to try to handle changing the profile:
https://github.com/mltframework/mlt/blob/master/src/framework/mlt_consumer.c#L208
But I have never tried that, I do not know if you are triggering that, and even so, I am sure that other services do not have a similar feature.

[ddennedy]

It is not sufficient to stop and restart the consumer to change the frame rate. You actually have to destroy and recreate all the services (producers, filters, consumers).

Now that you mention it, it was not the intention of the profile design in MLT to require that, but alas I see we are doing that in Shotcut. We use the xml consumer and producer to rebuild the graph. When doing so, you need to set the property no_profile to 1 on the xml consumer to prevent it from serializing the profile information. Also, in case you did not know, the design of mlt_profile is that you use the same profile object reference for all of the objects in a graph.

[ddennedy]

it was not the intention of the profile design in MLT to require that

After further consideration, what I said about simply restarting the consumer may only work in simple cases like playing a video. As soon as you start specifying time values in frame counts, a rebuild is required. To accommodate that, when serializing the XML, you can tell it to save with time values with millisecond precision by setting property time_format to clock on the xml consumer. Then, upon deserialization, it maps these times to frame counts based on the new profile frame rate.

A similar problem occurs with anything spatially specified using pixel values, which is many properties, and changing the resolution. Except, here we do not have a corresponding comprehensive solution. Many spatial properties also accept a percentage value (relative to profile resolution) but not all. However, you can preview at a lower resolution using something called preview scaling (see docs on web site), but keep in mind that is not perfect fidelity even though it comes close.