[HOLD] Adding method to sample "best" frames from a video

[brimoor]

Ports over work from https://github.com/voxel51/theta on adaptively sampling frames from video.

Specifically, provides a fiftyone.brain.sample_best_video_frames() method in the public brain interface with the following contract:

def sample_best_video_frames(
    video_path,
    out_frames_dir,
    target_num_frames=None,
    target_accel=None,
    target_fps=None,
    size=None,
    max_size=None,
):
    """Adaptively samples the best frames from the input video.

    The "best" video frames at a given sampling density are defined as the
    frames with highest image quality that are most representative of the
    visual content in the video.

    Provide one of ``target_num_frames``, ``target_accel``, or ``target_fps``
    to perform the sampling.

    Args:
        video_path: the path to the video to process
        out_frames_dir: a directory to write the sampled frames
        target_num_frames (None): the target number of frames to sample
        target_accel (None): a desired target acceleration factor to apply when
            sampling frames. For example, a target acceleration of 2x would
            correspond to sampling every other frame, on average
        target_fps (None): a desired target sampling rate, which must be less
            than the frame rate of the input video
        size (None): a desired output ``(width, height)`` for the sampled
            frames. Dimensions can be -1, in which case the input aspect ratio
            is preserved. By default, the input frame size is maintained
        max_size (None): a maximum ``(width, height)`` allowed for the sampled
            frames. Frames are resized as necessary to meet this limit, and
            ``size`` is decreased (aspect-preserving) if necessary to satisfy
            this constraint. Dimensions can be -1, in which case no limit is
            applied to them. By default, no maximum frame size is imposed

    Returns:
        a dict mapping frame numbers to the paths to the sampled frames in
        ``out_frames_dir``
    """

This is a precursor to another method that will sample frames dynamically across a collection videos.

Another feature that I want to add is to control how to sample frames within each equal-motion bin. Currently the highest quality frame is always selected. But a representative dataset for training needs to also include lower quality images. Thinking about the best way to expose this. Maybe the user gets to select between high quality, low quality, and random?

[jasoncorso]

Cool. I'll make an effort to look later. Any thoughts on how to substantiate the utility of this angle?

[ehofesmann]

target_num_frames isn't quite working right. I tried to run

fob.sample_best_video_frames(
    "~/fiftyone/quickstart-video/data/0587e1cfc2344523922652d8b227fba4-000014-video_052.mp4", 
    "test_video_frames/", 
    target_num_frames=4
)

It returned 6 frames instead of 4.

[brimoor]

target_num_frames isn't quite working right. I tried to run

fob.sample_best_video_frames(
    "~/fiftyone/quickstart-video/data/0587e1cfc2344523922652d8b227fba4-000014-video_052.mp4", 
    "test_video_frames/", 
    target_num_frames=4
)

It returned 6 frames instead of 4.

Oops good catch, there was a parentheses bug that has been there since THETA (just fixed it there too) that was always causing the first and last frames to be sampled.

Now it works as expected!

[ehofesmann]

Nice! Works great.

To answer your question about how to let a user get lower quality frames, one way to do it may be to let them select the percentage of frames that are selected by random and all others by high quality.

[brimoor]

Hmm that could an option, yes. I was also thinking about allowing the user to specify a quality percentile, and the algorithm would sample the frame w/ quality at the given percentile within each window. The current behavior would correspond to 100 percentile.

[ehofesmann]

Hmm that could an option, yes. I was also thinking about allowing the user to specify a quality percentile, and the algorithm would sample the frame w/ quality at the given percentile within each window. The current behavior would correspond to 100 percentile.

Ahh I think your suggestion is better. My suggestion would provide varying quality frames WITHIN a video. But it makes more sense to do what you said and let the user make videos where the quality varies between them but is similar for all frames within the same video.