Skip to content

CVNext/MPM

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

MPM: A Unified 2D-3D Human Pose Representation via Masked Pose Modeling

MPM: A Unified 2D-3D Human Pose Representation via Masked Pose Modeling
Zhenyu Zhang*, Wenhao Chai*, Zhongyu Jiang, Tian Ye, Mingli Song, Jenq-Neng Hwang, Gaoang Wang
arXiv 2023.

Install

pip install torch matplotlib

Dataset preparation

Our model is evaluated on Human3.6M and MPI-INF-3DHP datasets, and we ese AMASS dataset for better pre-training.

Human3.6M and MPI-INF-3DHP

Dataset setting is same as this repo P-STMO. You can download the processed .npz file from their repo and put the .npz files in ./dataset folder.

AMASS

coming soon

Evaluating our models

Checkpoint: link

evaluate on Human3.6M (CPN)

python trainer.py -f 243 --n_joints 17 --gpu 0,1 --reload 1 --layers 4 -tds 2 --previous_dir x.pth --refine --refine_reload 1 --previous_refine_name x_refine.pth

evaluate on Human3.6M (GT)

python trainer.py -f 243 k gt  --n_joints 17 --gpu 0,1 --reload 1 --layers 4 -tds 2 --previous_dir x.pth --refine --refine_reload 1 --previous_refine_name x_refine.pth

evaluate on MPI_INF_3DHP(GT)

python trainer_3dhp.py -f 243 --n_joints 16 --gpu 0,1 --reload 1 --layers 4 -tds 1 --previous_dir x.pth --refine --refine_reload 1 --previous_refine_name x.pth

Pretraining from scratch

Prepare Poseaug Generator

You should follow the instructions in poseaug and got generator checkpoint for human3.6M. Then put the generator checkpoints in ./Augpart/chk foler. You can put as many as you can get and modify the list in file ./Augpart/gan_preparation.py

Pretrain a model for 17 joints (only on h36m dataset)

python pretrainer.py --MAE -f 243 --train 1 -k gt --n_joints 17 -b 1024 -tds 2 --layers 4 --dataset h36m --lr 0.0001 -lrd 0.97 -tmr 0.6 -smn 5 --gpu x,y --name task_name 

Pretrain a model for 16 joints (only on h36m dataset with poseaug)

python pretrainer.py --MAE -f 243 --train 1 -k gt --n_joints 16 -b 1024 -tds 2 --dataset h36m --lr 0.0001 -lrd 0.97 --layers 4 -tmr 0.6 -smn 5 --gpu x,y --name task_name 

Pretrain a model for 16 joints (on multiple dataset)

python pretrainer.py --MAE -f 243 --n_joints 16 -b 1024 -k gt -tds 2 --train 1 --dataset h36m,3dhp,amass --layers 3 --lr 0.0001 -lrd 0.97 -tmr 0.6 -smn 5 --gpu x,y --name task_name 

Train HPE Task on h36m from scratch

N_JOINTS x and Layers n hould keep consistent with the pre-trained model.

python trainer.py -f 243 -k gt --train 1 --n_joints x -b 1024 --gpu 0,1 --lr 0.0007 -lrd 0.97  --layers 4 -tds 2 (--MAEreload 1 --previous_dir /path/to/pretrainedcheckpoint)(optional)

After training on human3.6M dataset, you can refine the model by:

python trainer.py -f 243 -k gt --train 1 --n_joints x -b 1024 --gpu 0,1 --lr 0.0001 -lrd 0.97  --layers 4 -tds 2 --reload 1 --previous_dir /path/to/bestcheckpoint --refine

Train 3DHPE on mpi_inf_3dhp from scratch

Finetune 3DHPE Model for MPI_INF_3DHP with 16 joints:

python trainer_3dhp.py -f 243 -k gt --train 1 --n_joints 16 -b 1024 --gpu 0,1 --lr 0.0007 -lrd 0.97 --layers 3 -tds 1 (--MAEreload 1 --previous_dir /path/to/pretrainedcheckpoint)(optional)

Train imcomplete 2D->3D Model

You can reload pretrained model or train model without reloading checkpoint:

python pretrainer.py -f 27 -b 2048 --model MAE -k gt --train 1 --layers 3 -tds 2 --lr 0.0002 -lrd 0.97 --name maskedliftcam -tmr 0 -smn 6 --gpu 0,1 --dataset h36m --MAE --comp2dlift 1 
(--MAEreload 1 --MAEcp /path/to/model)(optional)  

Train imcomplete 3D -> 3D Model

You can reload pretrained model or train model without reloading checkpoint:

python pretrainer.py -f 27 -b 2048 --model MAE -k gt --train 1 --layers 3 -tds 2 --lr 0.0002 -lrd 0.97 --name comp3dcam -tmr 0 -smn 3 --gpu 0,1 --dataset h36m --MAE --comp3d 1 
(--MAEreload 1 --MAEcp /path/to/model)(optional)

Acknowledgement

Our code refers to the following repositories.

We thank the authors for releasing their codes.

About

No description, website, or topics provided.

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • Python 100.0%