[CLIFF] misc questions for evaluation

[yohanshin]

Dear Authors,

Hi, I am trying to reproduce the evaluation table and have a few questions.

1. I could get similar (not exactly the same though) PA-MPJPE for 3DPW dataset using HR48 backbone. But I want to be confirmed if I am doing it correctly
   -- From the previous Issue post, I found that you are using the bbox obtained through projecting 3D keypoints onto image plane. In this case, which keypoints configuration did you use (e.g., COCO17, OpenPose25, H36M ...)
   -- I am using the camera focal length as a known value and computed it by averaging the x and y focal lengths. Is it same as what you've done?

2. Is the HR48 checkpoint uploaded in Google Drive the same as what you got for the Human36M evaluation performance?

Thank you so much for your answer in advance!

[zhihaolee]

1 a). We used the COCO17 keypoint configuration.
1 b). Yes, average of f_x and f_y.
2. No. Please see #138 (comment).

[yohanshin]

@zhihaolee

Thank you for your clarifications on the first and the second questions!

For the third one, I have a follow-up question. Does it mean that the checkpoint uploaded on Google Drive will show the MPJPE of 47.6 and PA-MPJPE of 32.7?

[zhihaolee]

Yes, you are right.

[RammusLeo]

@yohanshin Hi! Sorry to bother you. I'm trying to reproduce the result on Human3.6M with the released model hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pth, based on the H36M Protocol 2 3D keypoint annotation and the evaluation code released by SPIN. I have successfully got the best result on 3DPW, but I just got MPJPE = 58.2 and PA-MPJPE = 39.5, which is far away from the result given by the author. I wonder that have you reproduced the evaluation result on H36M? If so, what's your evaluation code and the test data? Thank you very much!

[yohanshin]

Hi @RammusLeo , I actually could not get the table for Human3.6M dataset. Sorry for not being so helpful..

[RammusLeo]

Thank you all the same for your response! I'll try other methods.

[ShamLich]

@yohanshin Hi! Sorry to bother you. I'm trying to reproduce the result on Human3.6M with the released model hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pth, based on the H36M Protocol 2 3D keypoint annotation and the evaluation code released by SPIN. I have successfully got the best result on 3DPW, but I just got MPJPE = 58.2 and PA-MPJPE = 39.5, which is far away from the result given by the author. I wonder that have you reproduced the evaluation result on H36M? If so, what's your evaluation code and the test data? Thank you very much!

hi @RammusLeo ,i got the same result as yours,I feel confused

[ShamLich]

@yohanshin Hi! Sorry to bother you. I'm trying to reproduce the result on Human3.6M with the released model hr48-PA43.0_MJE69.0_MVE81.2_3dpw.pth, based on the H36M Protocol 2 3D keypoint annotation and the evaluation code released by SPIN. I have successfully got the best result on 3DPW, but I just got MPJPE = 58.2 and PA-MPJPE = 39.5, which is far away from the result given by the author. I wonder that have you reproduced the evaluation result on H36M? If so, what's your evaluation code and the test data? Thank you very much!

are these numbers?@RammusLeo
MPJPE: 58.211485621217186
Reconstruction Error: 39.58124800903086