Effects in OTIO

[jminor]

Currently OTIO has very limited support for effects. What other formats, specifications, schemas, and systems for interchange of audio & video effects exist out there?

OpenFX has come up a few times in conversation: http://openeffects.org/
Are there others we should research?

How do FCPXML, EDL and AAF handle audio/video effects?

[jminor]

We briefly met representatives of OpenFX today (hi!) and the pointed us to this set of open source OpenFX plugins which cover a wide range of standard effects: https://github.com/tuttleofx/TuttleOFX

This might help in our investigation & understanding of the OpenFX ecosystem, but also might be useful for offline rendering of an OTIO with effects into a video file.

I wonder, @darbyjohnston have you looked at OpenFX?

[darbyjohnston]

I haven't but I was hoping to take a closer look soon. The TuttleOFX project looks interesting though unfortunately the GitHub hasn't been updated for a few years and it looks like their domain isn't working.

[apetrynet]

I believe @fabiencastan is/was involved in TuttleOFX. He is also on the ASWF slack so still in the industry I guess :)

[fabiencastan]

Hello @apetrynet ,
Yes, I'm still in the industry and still involed in OSS.
I was deeply involved in TuttleOFX years ago, but the project is not maintained anymore as we did not managed to federate enough partners around it and the market has evolved. Also the project was completely based on OpenFX, which is a great standard but it is not updated regularly enough.
I would be happy to help if you have questions on openfx.

[apetrynet]

Thanks @fabiencastan!

[jminor]

FYI, I attended the OpenFX TSC Meeting today (2022-09-08) and we discussed a few ways that OTIO and OpenFX could collaborate.

1. Using OTIO to hand off OpenFX plugin info from app-to-app.
2. Time-interpolated plugin parameters. Currently OpenFX defers this to the host application.
3. Using OTIO to give an OpenFX plugin an in-memory view of the timeline it is operating in/on.

Clearly 1 and 2 overlap with OTIO's goals, and could lead to new OTIO schema for animated video effects parameters. 3 is an interesting potential use case for OTIO.

Perhaps a nice next step would be for someone to demo how OpenFX works to an OTIO audience, and to demo how OTIO's existing effects schema works to an OpenFX audience.

[jminor]

Also, for reference, here is the AAF Effects Protocol document, which describes how some effects are represented in AAF: https://www.amwa.tv/_files/ugd/f66d69_898ba231b4224b0280bb7ce9ce607a89.pdf

[markreidvfx]

If it helps, here is a collection of AAF files for all the effects in a stock version of media composer.
https://github.com/markreidvfx/aaf_filter_samples

[jminor]

Wow - there's a lot in there!

I wonder if there's a subset of these effects which are in both these AAF samples, and in these OpenFX example plugins which we could use to form a concrete use case.

[barretpj]

Serialising an OFX plugin's state in a host-independent manner should be pretty straightforward, it consists of

• a triple of {identifier string, version number, context} e.g. {"com.example.SuperEffects.Wipe", 3, Transition}
• the value of each parameter, keyed by name (the difficulty here is that any animation of parameters is entirely left up to the host application. So we'd need to define at least a way of serialising keyframe time and values and some indication of the interpolation mode)
• the upstream node (apologies if there's a better OTIO name here) which is connected to each of the plugin's named input clips, if this isn't already explicit in the surrounding OTIO structure

[ssteinbach]

If it requires serializing keyframe time/values, this would depend on:

#331

[Dithermaster]

I have thoughts on this. I've helped implement OpenFX in two hosts and associated first-party effects and I'm active in the OpenFX community. I'm passingly familiar with OpenTimelineIO.

OpenFX defines the host <-> plugin API. It does not specify any storage or persistence. However, some years ago I proposed an XML format for describing a plugin chain including the parameter values (static or over time). It was based on something we had done in one of our hosts for effect "presets" which could be applied in a project. It never moved forward within OpenFX, but I think an OpenFX / OpenTimelineIO collaboration would benefit both groups, and would define not only a way to include effects in an OpenTimelineIO project, but by using a subset / subtree of OpenTimelineIO it would enable a way to describe plugin chains that could be moved between host applications (even if they don't fully support OTIO, although nicer if they did).

I fully support such a project and am happy to contribute ideas and feedback. However, since I don't have a horse in the race right now I can't contribute specification or code.

In addition to timeline or project interchange between similar hosts, I think the real workflow value is bridging between dissimilar applications. A specific use case that comes to mind is communicating effect chains from on-set pre-visualization applications to post-production applications (like editing and compositing). So in addition to comparatively simple adjustments like ASC-CDL, higher-end effects could be confidently added to the pipeline such as lens diffusion simulation or beyond-CDL color adjustments (such as those popular in HDR workflows).

Note that AMPAS ACES has some efforts in this area in S-2019-001 ACES Metadata File (AMF), which is worth looking at as well.

[meshula]

Hello @Dithermaster! That sounds interesting. Perhaps it might be a topic to bring up during an OpenTimelineIO TSC meeting ~ everyone is welcome to attend and propose topics.

[jminor]

@Dithermaster thanks for the pointer to AMF. I had not seen that before.

We can look at AMF as reference for general effect support, and also maybe as a direct integration similar to recent discussions about Framing Decision Lists (FDL) which have similar workflow considerations.

Skimming the AMF spec, it looks like it is a superset of Color Decision Lists (CDL), and includes ASC_SOP Slope, Offset, and Power values, which we already have limited support for in OTIO. So this is a good point of reference.

@barretpj I would love to see an more detailed example of how an OpenFX plugin's state could be serialized. If we had a proof of concept, or mockup showing how a set of non-animated OpenFX parameters could be carried in an OTIO that would help to clarify many open questions. Tackling animated parameters could be folded in later.

[garyo]

Correct -- this is intended to be a mockup of an actual effect graph storage format, so not just a single OFX effect. I assume that's what OTIO is looking for.

As to your point about inputs, that's valid for the General (multi-input) context. For the OTIO folks: Generators have no inputs. In a Filter or Retimer context there's only one input, and in a Transition there are two, with implicit ordering (outgoing clip first). Paint also has two, with implicit ordering (source, mask). So perhaps the inputs for the General context should be a dictionary instead, keyed by the input name defined by the effect -- which as @barretpj says, can be an arbitrary string:

  inputs: {
    'first source': generator,
    'the mask!': other_effect
 }

Although I do think most (all?) hosts will present the inputs in order of definition, I don't think the spec requires that.

[barretpj]

Optional inputs mean there's ambiguity in a simple list, unless you have a way of specifying the unconnected state. I'd say relying on order of definition is weak, for example because a plugin could easily define clips in a different order to different hosts if it wanted to, maybe to make their UI look better in that host. All OFX clip access is done by name, not by index.
I'd also argue for using an inputs dictionary everywhere, not making the General context special.

[garyo]

I agree. Since General context is the only one where optional inputs are allowed, this spec should always require a dictionary for inputs in that context. Also, since the OpenFX spec mandates particular input names for the other contexts (e.g. SourceFrom and SourceTo for transitions) it would be OK with me to use a dictionary keyed by input name in all contexts (rather than an array), if that's simpler or more future-proof.

I assume OTIO will want this spec to work for non-OFX effects; if some other API used an array of inputs rather than names, they could just be called input-1, input-2 in this saved form (as one example).

[garyo]

Also, though not particularly relevant to this conversation, an OFX input (Clip) can have a Label which should be shown in the UI, and can be distinct from the name.

[jminor]

This is very helpful, and becoming more clear to me as you talk about the details. The distinction between Generators, Filters, and Transitions should map well to OTIO concepts. We already have the naïve notion of a generator as a media reference for a clip which could evolve to align with the OpenFX concept.

[jminor]

At today's OpenFX meeting we looked briefly at the OTIO effects metadata that DaVinci Resolve outputs. It uses an application-specific metadata block to store effects parameters, keyframes, etc. This is great to see and could be a starting point for a generalized official schema for effects.

Here's a screenshot showing an example:

[garyo]

One question is how different interpolation types should be represented. How far down that rabbit hole should OTIO go? One possibility would be just "Hold", "Linear", and "Spline" (cubic spline knots? interpolating points with optional in/out tangents?) Also ensure that keyframes are time-based, not frame-based (or at least make that choice explicitly). That gets complicated in a retimer context.

It would be good to have the inputs explicitly represented as in my example above. Also at least for OpenFX, context (generator, filter, transition, general) is important.

For OpenFX, please use the effect's kOfxParamPropScriptName since that is more stable than the user-visible name.

I think each effect will need some kind of unique ID so it can be referenced by downstream effects (not sure how OTIO handles graphs at the moment). See my example.

I would not suggest including "Default Parameter Value" -- but I would suggest always including all param values even if they are at the default. This is robust against the effect changing its defaults in a new version, breaking the comp.