Source code of the CVPR'2022 paper "Thin-Plate Spline Motion Model for Image Animation"
PS: The paper trains the model for 100 epochs for a fair comparison. You can use more data and train for more epochs to get better performance.
- Integrated into Huggingface Spaces 🤗 using Gradio. Try out the Web Demo:
- Try the web demo for animation here:
- Google Colab:
We support python3
.(Recommended version is Python 3.9).
To install the dependencies run:
pip install -r requirements.txt
There are several configuration files one for each dataset
in the config
folder named as config/dataset_name.yaml
.
See description of the parameters in the config/taichi-256.yaml
.
-
MGif. Follow Monkey-Net.
-
TaiChiHD and VoxCeleb. Follow instructions from video-preprocessing.
-
TED-talks. Follow instructions from MRAA.
Here are VoxCeleb, TaiChiHD and TED-talks pre-processed datasets used in the paper. Baidu Yun Download all files under the folder, then merge the files and decompress, for example:
cat vox.tar.* > vox.tar
tar xvf vox.tar
To train a model on specific dataset run:
CUDA_VISIBLE_DEVICES=0,1 python run.py --config config/dataset_name.yaml --device_ids 0,1
A log folder named after the timestamp will be created. Checkpoints, loss values, reconstruction results will be saved to this folder.
To train a model on specific dataset run:
CUDA_VISIBLE_DEVICES=0 python run.py --mode train_avd --checkpoint '{checkpoint_folder}/checkpoint.pth.tar' --config config/dataset_name.yaml
Checkpoints, loss values, reconstruction results will be saved to {checkpoint_folder}
.
To evaluate the reconstruction performance run:
CUDA_VISIBLE_DEVICES=0 python run.py --mode reconstruction --config config/dataset_name.yaml --checkpoint '{checkpoint_folder}/checkpoint.pth.tar'
The reconstruction
subfolder will be created in {checkpoint_folder}
.
The generated video will be stored to this folder, also generated videos will be stored in png
subfolder in loss-less '.png' format for evaluation.
To compute metrics, follow instructions from pose-evaluation.
- notebook:
demo.ipynb
, edit the config cell and run for image animation. - python:
CUDA_VISIBLE_DEVICES=0 python demo.py --config config/vox-256.yaml --checkpoint checkpoints/vox.pth.tar --source_image ./source.jpg --driving_video ./driving.mp4
The main code is based upon FOMM and MRAA
Thanks for the excellent works!
And Thanks to:
-
@chenxwh: Add Web Demo & Docker environment
-
@TalkUHulk: The C++/Python demo is provided in Image-Animation-Turbo-Boost
-
@AK391: Add huggingface web demo