Improve handling of rates when converting to timecode strings

[jminor]

This PR improves OTIO's to_timecode, is_valid_timecode_rate, and nearest_valid_timecode_rate functions so they handle rate values which are close, but not exactly matching the correct rates.

Specifically the previous strategy of keeping a table of commonly mis-typed non-integer rates (e.g. 29.97, 29.976, 23.98, etc.) is replaced by a heuristic which matches rates to the closest correct rate within some tolerance.

Rates checked by is_valid_timecode_rate and nearest_valid_timecode_rate now match each other, and adhere to ST 12-1:2014 - SMPTE Standard - Time and Control Code (which is now freely available 👏🎉)

We also spot-checked some comparisons between Avid Media Composer and OTIO to make sure we got the drop frames in the right spot - that was already correct before this PR.

Note: there is 1 test failing which needs to be addressed, and my updates to the other tests should be scrutinized to make sure the changes are good.

For the failing test... something doesn't match up.

• Avid Media Composer's Calculator says: 1084319 frames = 05:01:11:29 @ 60 fps
• OpenTimelineIO says: 1084319 frames = 05:01:11:59 @ 60 fps
• https://robwomack.com/timecode-calculator/ says: 1084319 frames = 05:01:30:03 @ 60 fps

Test code: otio.opentime.from_frames(1084319, 60).to_timecode()

[meshula]

To summarize, MC and OTIO agree disagree, but time-code-calculator significantly differs agrees with OTIO. ? Maybe we need to ping Avid. the calculator's author...

[markreidvfx]

I would like to suggest RationalTime::from_timecode explicitly take the nominal fps as a argument along with the frame rate

like what I proposed here: #1452 (comment)

RationalTime
RationalTime::from_timecode(std::string const& timecode, 
                            uint32_t           timecode_fps,
                            double             rate,
                            ErrorStatus*       error_status)

This would eliminate the need to do all the guessing from the playback frame rate. 
You might for example be using 24 frame timecode at playback rate of 30 fps, which isn't possible with the current method.

[markreidvfx]

I tried that online tc calculator.
Its pretty finicky but you might have had it set to 59.97fps DF , I get
1084319 frames = 05:01:11:59 @ 60 fps.
1084319 frames = 05:01:11:59 @ 59.97 fps. Non Drop Frame (should be the same as 60)
and
1084319 frames = 05:01;30;03 @ 59.97 fps Drop Frame

That rate should really be 59.94, not 59.97, but oh well.
I haven't figure out how to get 59.94 DF TC working on my avid yet to verify if that calculation is correct.

[jminor]

MC and OTIO are off by 30 frames from each other.

[markreidvfx]

The TC display in MC is might be showing 30fps TC where it duplicates every step. My MC cannot display 60 TC without using a burn in.

To make things a little more odd, the 60fps TC burnin only seems to works when working at 59.94fps. I am still using 2018, maybe there is better support in a newer version.

[markreidvfx]

ok, in MC general settings there is option to change the TC display format from 30fps to 60 fps

Still haven't found a 60fps drop frame option.

[jminor]

Here's the Media Composer features I used to check against:

• The "=" menu in the bottom left corner of the timeline window has "Show Track" which lets me turn on multiple timecode rulers along the timeline:
  

• The Timecode Window right-click menu lets me choose these:
  

• The Calculator Window lets you convert between "Total Frame Count" and various rates. This is where I got 1084319 frames = 05:01:11:29 @ 60 fps
  

[jminor]

@markreidvfx can you explain more about using 24 frame timecode with 30 fps playback? That's not a scenario I've seen, so I'm confused about how it works.

With your proposed change, would these produce the same answer?

a = otio.opentime.from_timecode(timecode_str, 24, 30)
b = otio.opentime.from_timecode(timecode_str, 24, 24).rescaled_to(30)

and the other direction:

a = otio.opentime.from_frames(frame_number, 30).rescaled_to(24).to_timecode()
b = otio.opentime.from_frames(frame_number, 30).to_timecode(rate=24)

[markreidvfx]

No, those would produce different answers. maybe using 23.97 is a bit more clear.
It would be the equivalent of this

frame = otio.opentime.from_timecode(timecode_str, 24).value
a = otio.opentime.from_frames(frame, 23.97)
b = otio.opentime.from_timecode(timecode_str, 24).rescaled_to(23.97)

a.to_seconds() != b.to_seconds()

Timecode is converted to a frame number first.
Then that frame number is converted to a time at current frame rate.

Maybe adding another arg isn't the correct answer. The big thing I'm trying to get across is that timecode should be treated as frame number encoding and is not always reliable to be a timestamp. It should really be called framecode in my opinion. :p

This is why I normally use a pattern like this

frames_to_seconds(timecode_to_frames(timecode_str, timecode_fps), frame_rate)

and the other direction:

frames_to_timecode(seconds_to_frames(seconds, frame_rate), timecode_fps, drop=False)

[markreidvfx]

what is the the project format/edit rate?

some formats can't access certain timecodes.

[jminor]

Aha! That explanation makes total sense now. The conversion to a frame count vs a timestamp really makes it clear - thanks @markreidvfx !

I'll check my MC project settings tomorrow.

I'd love to compare this to Resolve and/or Premiere as well. Are there other trusted timecode conversion tools or software libraries out there to compare to?

[jminor]

This PR overlaps or conflicts with this older PR: #1180

[meshula]

@jminor I closed that one in favor of this one, since this one rectifies the treatments of documented SMPTE rates with the code. @splidje NLEs have various "tricks" for dealing with high frame rates, let's carry on the discussion of what to do with the high frame rates here.

[markreidvfx]

I've verified those calculations in Resolve, it actually supports 59.94 TC with drop frame.

1084319 frames = 05:01:11:59 @ 59.94 fps without drop frame
1084319 frames = 05:01;30;03 @ 59.94 fps with Drop Frame

You can set the tc format when you create a timeline or in your project settings

You can toggle between frames and tc by right clicking on the timecode in the viewer

[markreidvfx]

Resolve also additionally supports a 16, 18, 47.952 and 48 TC format.
The 47.952 appears to just be 48, like 23.976 is just 24

[meshula]

Thanks for doing the extra checks! So I think the theory is that MC is displaying time code at 30, even for a selected rate of 60 (as opposed to MC displaying a value that is 30 frames off of our calculation). A check for that would be to knock off 5 frames at 30, and check if MC then shows 24 instead of 29, or if it shows 19. 24 would demonstrate it is displaying a 30 rate, instead of 60; whereas 19 would show that the calculation is off by 30.

[markreidvfx]

I'm quite sure @jminor had his avid set in 30 TC mode for 60. Its the default setting. The timecode ticks every 2 frames and the top dot of the last colon blinks on and off to tell you if your on an even or odd frame.

timecode.mp4

This mode is somewhat described in the spec intro, page 5

Progressive video systems with frame rates above 30 frames per second are described in this document, documenting what have become “de facto” implementations. Since the frame rate of these 50 and 60 frames-per-second progressive systems exceeds the frame count capacity of the time address, counting is done on frame pairs, which results in an edit resolution of two frames using traditional linear time code.

There are some further details in 12.1

[markreidvfx]

I've also verified those timecode values in Unreal Engine

60 fps NDF

59.94 fps DF

[jminor]

Sorry for the delay in getting back to this. I should hopefully have time to revisit this in a couple of weeks.

In the meantime, I happened across this which is interesting to compare/contrast with OTIO's approach: https://github.com/orchetect/TimecodeKit

[meshula]

Had a look at the link you provided, quite an interesting read ~ TimeCodeKit's major difference, I'd say, is that it introduces a strongly typed TimeCode object from which a string can be generated, and which functions as a mathematical object. It hasn't got a stronger ability than otio::RationalTime to represent a point in time, or a stronger ability to do math than otio::RationalTime, and would probably resist computations we are interested in, such as linear timewarps.

Another point of contrast, philosophically, is the view suggested by @markreidvfx earlier in the thread, that the timecode string should be considered, in a way, a "rendered" interpretation of a time, as opposed to a ground truth representation of time.

TimeCodeKit does some interesting things that we don't support, at all. If you decrement time below zero, it will wrap around to 24 hours. We generate negative values. We could possibly do something similar in genarating time code strings. At the moment, negative time values are rejected. If the are negative and greater than -24 hours, we could wrap around.

I think it would be worth working through the math of TimeCodeKit to determine whether our conversion algorithm matches. Also, it would be worth working through the unit tests, and perhaps adding one to one corresponding unit tests to our own code.

[meshula]

I ran the tests in my own repo, the python unit tests are what failed, the tests themselves may need correction.

======================================================================
FAIL: test_invalid_rate_to_timecode_functions (test_opentime.TestTime.test_invalid_rate_to_timecode_functions)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/OpenTimelineIO/OpenTimelineIO/tests/test_opentime.py", line 400, in test_invalid_rate_to_timecode_functions
    with self.assertRaises(ValueError):
AssertionError: ValueError not raised

======================================================================
FAIL: test_timecode_infer_drop_frame (test_opentime.TestTime.test_timecode_infer_drop_frame)
---------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/runner/work/OpenTimelineIO/OpenTimelineIO/tests/test_opentime.py", line 347, in test_timecode_infer_drop_frame
    self.assertEqual(t.to_timecode(rate, drop_frame=None), timecode)
AssertionError: '05:01:30;03' != '05:01:11;59'
- 05:01:30;03
+ 05:01:11;59

[jminor]

Yes, @meshula that failing test is the one I mentioned in the initial post at the very top of this thread. My question is/was: what is the correct return value for otio.opentime.from_frames(1084319, 60).to_timecode() ?

[meshula]

otio.opentime.from_frames(1084319, 60).to_timecode() # no dropframe rate requested

referencing @markreidvfx's reported values from Resolve and Unreal

1084319 frames = 05:01:11:59 @ 59.94 fps without drop frame
1084319 frames = 05:01:30;03 @ 59.94 fps with Drop Frame

suggest that answer should be 05:01:11:59

I've gone through this thread a bunch of times in the last hour, and I feel we can be confident about that.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 89.65517% with 6 lines in your changes missing coverage. Please review.

Project coverage is 81.63%. Comparing base (c0e97b0) to head (f8aaf84).
Report is 28 commits behind head on main.

Files with missing lines	Patch %	Lines
src/opentime/rationalTime.cpp	70.58%	3 Missing and 2 partials ⚠️
src/opentime/errorStatus.cpp	50.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1477      +/-   ##
==========================================
- Coverage   84.11%   81.63%   -2.49%     
==========================================
  Files         198      176      -22     
  Lines       22241    12683    -9558     
  Branches     4687     2793    -1894     
==========================================
- Hits        18709    10354    -8355     
+ Misses       2610     1789     -821     
+ Partials      922      540     -382     

Flag	Coverage Δ
py-unittests	81.63% <89.65%> (-2.49%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines	Coverage Δ
src/opentime/rationalTime.h	92.55% <ø> (-2.13%)	⬇️
...lineio/opentime-bindings/opentime_rationalTime.cpp	94.82% <100.00%> (+0.23%)	⬆️
tests/test_opentime.py	99.75% <100.00%> (+<0.01%)	⬆️
src/opentime/errorStatus.cpp	55.55% <50.00%> (+11.11%)	⬆️
src/opentime/rationalTime.cpp	73.94% <70.58%> (+2.34%)	⬆️

... and 118 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f40c509...f8aaf84. Read the comment docs.

[jminor]

Returning to this PR after a long hiatus... I rebased onto latest main, added some doc strings, fixed the bogus test case, and addressed some review notes from 2023.