New Feature

Support for rectangular videos of non-512x512 dimensions, with automatic adaptation to the appropriate resolution

Support for automatic adaptation of the output video's frame rate

Support for additional input prompt to guide video output

Support for utilizing OpenPose to guide video output

🦒 Colab

Colab	Info
	magic_animate_colab
	vid2densepose vid2openpose

MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model

Zhongcong Xu · Jianfeng Zhang · Jun Hao Liew · Hanshu Yan · Jia-Wei Liu · Chenxu Zhang · Jiashi Feng · Mike Zheng Shou


National University of Singapore   |   ByteDance

📢 News

• [2023.12.8] Add loading local safetensors or ckpt,you can change config/prompts/animation.yaml about pretrained_model_path for your local SD1.5 model. such as "D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"
• [2023.12.4] Release inference code and gradio demo. We are working to improve MagicAnimate, stay tuned!
• [2023.11.23] Release MagicAnimate paper and project page.

⚒️ Installation

prerequisites: python>=3.8, CUDA>=11.3, ffmpeg and git.

Python and Git:

• Python 3.10.11: https://www.python.org/ftp/python/3.10.11/python-3.10.11-amd64.exe

• git: https://git-scm.com/download/win

• Install ffmpeg for your operating system (https://www.geeksforgeeks.org/how-to-install-ffmpeg-on-windows/)

  notice:step 4 use windows system Set Enviroment Path.

Give unrestricted script access to powershell so venv can work:

• Open an administrator powershell window
• Type Set-ExecutionPolicy Unrestricted and answer A
• Close admin powershell window

Install with Powershell run install.ps1 or install-cn.ps1(for Chinese)

###Use local model

Add loading local safetensors or ckpt,you can change config/prompts/animation.yaml about pretrained_model_path for your local SD1.5 model. such as "D:\\stablediffusion-webui\\models\Stable-diffusion\\v1-5-pruned.ckpt"

🎨 Gradio Demo

Online Gradio Demo:

Try our online gradio demo quickly.

Local Gradio Demo:

Launch local gradio demo on single GPU:

Powershell run with run_gui.ps1

Launch local gradio demo if you have multiple GPUs:

Edit run_gui.ps1 set $mutil_gpu=1 then run.

Then open gradio demo in local browser.

🙏 Acknowledgements

We would like to thank AK(@_akhaliq) and huggingface team for the help of setting up oneline gradio demo.

🎓 Citation

If you find this codebase useful for your research, please use the following entry.

@inproceedings{xu2023magicanimate,
    author    = {Xu, Zhongcong and Zhang, Jianfeng and Liew, Jun Hao and Yan, Hanshu and Liu, Jia-Wei and Zhang, Chenxu and Feng, Jiashi and Shou, Mike Zheng},
    title     = {MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model},
    booktitle = {arXiv},
    year      = {2023}
}