Non-linear Motion Estimation for Video Frame Interpolation using Space-time Convolutions

• ArXiv (8 page version)
• Paper (4 page version)
• Supplementary
• Poster

ℹ️ Paper accepted at CLIC workshop @ CVPR 2022 !

Repo under construction!

Highlights

• We introduce a novel frame interpolation algorithm that utilizes both flow and occlusion maps between four input frames to estimate an automatically adaptable pixel-wise non-linear motion model to interpolate the frames.
• We propose a parameter and runtime-efficient 3D CNN named GridNet-3D to aggregate multi-scale features efficiently. Our GridNet-3D has only 2.44 M parameters which performs better than UNet-3D with 42.06 M parameters.

Requirements

• torch==1.1.0 (CUDA 10.1)
• torchvision==0.3.0
• opencv-python==3.4.2
• scikit-image==0.17.2

Setup

Please setup IRR repository and update installation directory in model.py.

Datasets

• Vimeo Septuplet
• DAVIS
• HD
• GoPro

The quintuplets used for evaluation are stored in datasets folder as .csv files. Please modify the absolute path accordingly.

Inference

python eval.py --dataset <dataset name> --data_root <dataset location>

References

Our code is built upon the following existing papers and repositories.

• QVI

• IRR

• FLAVR

• SuperSloMo

Citation

@InProceedings{Dutta_2022_CVPR,
    author    = {Dutta, Saikat and Subramaniam, Arulkumar and Mittal, Anurag},
    title     = {Non-Linear Motion Estimation for Video Frame Interpolation Using Space-Time Convolutions},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops},
    month     = {June},
    year      = {2022},
    pages     = {1726-1731}
}

Contact:

<github username>[email protected]