nicholasli1995 / evoskeleton

Official project website for the CVPR 2020 paper (Oral Presentation) "Cascaded deep monocular 3D human pose estimation wth evolutionary training data"

EvoSkeleton

This is the project website containing relevant files for the CVPR 2020 paper "Cascaded deep monocular 3D human pose estimation with evolutionary training data". The usage and instructions are organized into several parts serving distinct purposes. Please visit the corresponding sub-page for details. For Q&A, go to discussions. If you believe there is a technical problem, submit to issues.

News:

(2021-04-08): Release v-1.0. The support for pre-trained models is strengthened. More details have been added to the supplementary material.

Cascaded 2D-to-3D Lifting

This sub-page details how to train a cascaded model to lift 2D key-points to 3D skeletons on H36M.

If you do not want to prepare synthetic data and train the model by yourself, you can access an examplar pre-trained model here and follow the instructions in the document. This model can be used for in-the-wild inference as well as reproducing the results on MPI-INF-3DHP. The evaluation metric for MPI-INF-3DHP can be accessed here.

Performance on H36M (Link to pre-trained models)

Protocol #1	Avg.	Dir.	Disc	Eat	Greet	Phone	Photo	Pose	Purch.	Sit	SitD.	Smoke	Wait	WalkD.	Walk	WalkT.
Martinez et al. (ICCV'17)	62.9	51.8	56.2	58.1	59.0	69.5	78.4	55.2	58.1	74.0	94.6	62.3	59.1	65.1	49.5	52.4
Ours (S15678)	49.7	45.6	44.6	49.3	49.3	52.5	58.5	46.4	44.3	53.8	67.5	49.4	46.1	52.5	41.4	44.4

Protocol #2	Avg.	Dir.	Disc	Eat	Greet	Phone	Photo	Pose	Purch.	Sit	SitD.	Smoke	Wait	WalkD.	Walk	WalkT.
Martinez et al. (ICCV'17)	47.7	39.5	43.2	46.4	47.0	51.0	56.0	41.4	40.6	56.5	69.4	49.2	45.0	49.5	38.0	43.1
Ours (S15678)	37.7	34.2	34.6	37.3	39.3	38.5	45.6	34.5	32.7	40.5	51.3	37.7	35.4	39.9	29.9	34.5

Hierarchical Human Representation and Data Synthesis

This sub-page gives instructions on how to use the 3D skeleton model and how the evolution algorithm can be used to discover novel data.

2D Human Pose Estimation on H3.6M

This page shows how to perform 2D human pose estimation on Human 3.6M dataset with the pre-trained high-resolution heatmap regression model. The highly accurate 2D joint predictions may benefit your 3D human pose estimation project.

Method	Parameters	FLOPs	Average Joint Localization Error (pixels)
CPN (CVPR' 18)	-	-	5.4
Ours (HRN + U + S)	63.6M	32.9G	4.4

Dataset: Unconstrained 3D Pose in the Wild

This sub-page describs the newly collected dataset Unconstrained 3D Human Pose in the Wild (U3DPW) and gives instructions on how to download it.

Interactive Annotation Tool

This sub-page provides usage of an annotation tool that can be used to label 2D and 3D skeleton for an input image. U3DPW was obtained created with this tool and this tool may help increasing the scale of 3D annotation for in-the-wild images.

Environment

• Python 3.6
• Numpy 1.16
• PyTorch 1.0.1
• CUDA 9

For a complete list of other python packages, please refer to spec-list.txt. The recommended environment manager is Anaconda, which can create an environment using the provided spec-list. Certain tool in this project may need other specified environment, which is detailed in its corresponding page.

License

A MIT license is used for this repository. However, certain third-party dataset (Human 3.6M) and tool (SMPLify) are subject to their respective licenses and may not grant commercial use.

Citation

Please star this repository and cite the following paper in your publications if it helps your research:

@InProceedings{Li_2020_CVPR,
author = {Li, Shichao and Ke, Lei and Pratama, Kevin and Tai, Yu-Wing and Tang, Chi-Keung and Cheng, Kwang-Ting},
title = {Cascaded Deep Monocular 3D Human Pose Estimation With Evolutionary Training Data},
booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}

Link to the paper: Cascaded Deep Monocular 3D Human Pose Estimation With Evolutionary Training Data

Link to the oral presentation video: Youtube

evoskeleton's Issues

Do you plan to publish part of the HRnet code?

Do you plan to publish part of the HRnet code?

Result is a bit strange

Follow your guide, I generated some new data and trained the model with the data.
here is the command:
evolve python evolve.py -generate True -WS True -SS "S1"
train python 2Dto3Dnet.py -train True -num_stages 3 -ws True -ws_name "S1" -twoD_source "synthetic" -evolved_path "/home/hcy/project/3d_human_pose/EvoSkeleton-master/data/human3.6M/evolved/S1/evolved_data/generation_13.npy"
After data enhancement, two files were generated , generation_1.npy and generation_13.npy. I trained with the latter one. The result is a bit strange. After 200 epochs, you test the model with the test data.The picture below shows the test results.

I didn't change any code of your project. But the result is better than 50.5 which you showed on your paper .
I am curious why this happens.

Data Preprocessing when test on 3dhp dataset

After training the model on the H36m dataset, if we need to evaluate on the 3dhp dataset, how to preprocess the 3dhp dataset? Considering that there are differences in the definitions of joint for both datasets (no nose joint in 3dhp), should we delete the nose joint in H36m dataset during the model training process (Input size: 16 * 2 Output size: 15 * 3).
And there are two types of 3D joints coordinates in 3dhp dataset, which one should we use during evaluating, should we delete the nose joint (16 * 3), align the keypoints to the root joint and delete the root joint (15 * 3) ?
Should we first resize the scale of estimated 3dhp 2D coordinates and then normalizing the 2D keypoints using H36m stats? And what about the 3dhp 3D keypoints? Are there any other details that need to attention?

I can't find the model

Hello, after reading your code, I found that the model file which should under the libs is missing.

Resulted 2D pose of the mentioned HR-Net on Human 3.6M

Hi,

Thanks for your great work and code. I noticed that you released the pre-trained model of HR-Net for 2D pose estimator, but we still need the crop step, which may lead to an inconsistent 2D result. Could you please directly provide the estimated 2D pose (it may be in .npy format and shouldn't be too large) from the HR-Net on Human 3.6M? I think it will be very helpful for the community.

How to calculate stats for examples/inference.py? (e.g. mean_3d, std_3d dim_ignore_3d)

In examples/inference.py, how do I calculate stats passed to unNormalizeData() from evo_skeleton.dataset.h36m.data_utils import unNormalizeData, as I am using my own images, and can't create a new stats.npy for my images. I would eventually like to pass video to it too.

Thanks in advance

generating other poses from a known pose

Is there a way to use your code to generate all or comment poses from a given image?

how to inference with wild image

hello, I want to use this model to predict some my wild picture . what should I do. How to get these 2d info of my picture.

Question about U3DPW and H36M

Hi, thank you very much for your excellent job!

Could you please explain how to use U3DPW as an augmented dataset for H36M? The keypoints on the H36M dataset and the U3DPW dataset are different. The visualization results of the 3d skeleton on two datasets are not totally the same.

Thank you very much for your help!

Missing "data" & "resource" folder?

I do not see any folder named "data" or "resources" inside the EvoSkeleton folder for the separately downloaded constraints and Human3.6 folders to be in mentioned in https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/HHR.md

run question

hello,I want to ask Where can I see the data I evolved, and what exactly does the evolved_path refer to? (python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1" -twoD_source "HRN/synthetic" -evolved_path "YourDataPath")
Can you give me an example? Thank you very much!

AttributeError: 'NoneType' object has no attribute 'keys'

Hello, the following error occurred when I was running the training code. May I ask what the problem is and how to solve it?

Training with less joints

Hello again,

Sorry for another issue!

I was trying to modify your code to train with 13 joints (instead of 17) in fully supervised manner, so far it seems to run. There are a few things I would like to consult you: (1) should I use lesser number of stages and blocks? (2) should I train with lower learning rate or fewer epochs? (3) Will the removal of head, thorax and spine affect the performance a lot? (4) I can see that you remove nose in your training, may I know why?

I really appreciate your help! Thanks in advance!

Shiyang

camera params in cameras.npy

how to get cameras.npy, is it provided in the offical website?

training input size and inference input size not match

Hi, in your training scirpit h36m2Dpose.py, after loading the dataset, the input shape is (34, ), but in the inference.py, the input size is (32, ), why is that?

Query on Keypoint order for Inference in the wild

Hey, in the inference.py file, it reads the example_annot.npy for annotations. Can you please tell me which indices correspond to which keypoints? Thanks!

Hello ask a question

could you please let me know whether the code for Heatmap Regression Model(Stage 1) parts already exists with the provided code？

Obtain 3D skeleton with 2D key-points as inputs using SMPLify by own data

Thank you so much for providing the code also now I met the problem when I want to obtain a 3D skeleton with 2D keypoint.
The error is like this when I run the code
"python fit_3d.py -dataset_dir anno -model_dir smplify/models/basicModel_neutral_lbs_10_207_0_v1.0.0.pkl"
Traceback (most recent call last):
File "fit_3d.py", line 99, in
main(opt)
File "fit_3d.py", line 62, in main
do_degrees=do_degrees
File "/home/panda/EvoSkeleton/libs/annotator/smplify/fit_3d.py", line 451, in run_single_fit
viz=viz)
File "/home/panda/EvoSkeleton/libs/annotator/smplify/fit_3d.py", line 120, in initialize_camera
center = np.array([img.shape[1] / 2, img.shape[0] / 2])
AttributeError: 'NoneType' object has no attribute 'shape'

It shows that it can not use my 2D input keypoint. Many thanks

Regarding the 2D HPE model

Hello,

I am trying to use your pre-trained 2D pose model to estimate 17 joints.
May I know which architecture do you use for it, Stacked Hourglass? Also, what data do you use to train the model, are they 2D joints from Human3.6m?

Thanks,
Shiyang

2D detection results on MPI_INF_3DHP dataset

Hi, thank you for your great work!

Could you please provide the pre-processed 2D detections and 3D poses for MPI_INF_3DHP test set. As such, we can evaluate the model generalizability under cross-dataset setup.

Thank you!

weird 3D Pose

I ran into a strange problem: lengthen the length of thigh and shorten the length of calf in the 2D Pose, the angle between the thigh and calf is different from normal in the output 3D pose.

I wonder why your model is "cascaded"

When I think about 'cascaded network', I come up with a network of modules connected in consecutive order, just like cascaded pyramid network(CPN).
In your implementation, every stage gets 2D skeleton inputs, and generates 3D skeleton estimations.
And the estimation loss is just the difference between the target and the sum of their estimations.
Therefore, I think that your 2D-to-3D estimation network is composed of multiple 2D-to-3D estimation models bagged together.
Of course, the gradient flows through every stage.
But the latter stage does not get the output of the former stage, so they are not connected sequentially.

TODO: Model Zoo for pre-trained models under different settings

It seems that the pre-trained model in examples.zip is weakly-supervised trained because it yields mpjpe of 50mm on GT 2D pose.
Where can I find the fully-supervised trained model, to reproduce the results 34.5 in Tab.3 in protocol P1*?

Thanks in advance.

2D Pose Estimation

Hi, Is there a possibility to do 2D pose estimation of an arbitrary image with the current code that is uploaded? the inference code is uploading skeleton from a fixed, predefined dictionary of annotation....
Thanks very much in advance
Best regards,
Leila

The best way to correct 3D pose for multi-person dataset

Hi,

Thanks for the great work.

I have a dataset contains multi person's 3D pose. Some of them are not correct so I wanted to use your tool to correct those pose.

I'm not sure the best way of using the tools. This is what I have in my mind:

1. Crop the single person image using the bounding box from a multi person image.
2. Since I already have the 2D pose annotations, I will skip the 2D annotation, but do the 3D parameter fitting.
3. Then I will do the 3D annotations to correct the pose.
4. Merge every single person's result to the multi-person image.

Something that's not clear in my mind:

In step 1, I have all the 2D and 3D annotations for multiple person. However, they are global. If I crop to a single person image, should I change the pose annotations to local? Also since I crop the image from the bounding box, the size of every image will be different. Will that be a problem? I'm not sure what is the best way of doing it. It seems a lot of works.

Thanks in advance for answering my questions!

eval on 3DHP

nice code, nice paper, nice job!
my question is :
Can you provide the code for evaluating the 3DHP datasets,(e.x. pck3d & AUC calculation) or share other repositories that have the eval. code

How to make 2d pose estimation output .npy?

Hello.
I want to run in the wild data using inference. but I don't know how to make 2d pose estimation output (.npy file).
According to what I understand, For making npy file, I should use h36m2Dpose.py or annotate_2D.p, right?

Can you explain how?

thank you.

Location of the hip/pelvis joint?

Does the 2d-to-3d lifting model actually predict a location for the hip/pelvis joint? OR, does the model predict all of the other 16 joints in relation to the hip joint?

It seems like when we run UnNormalizeData in examples/inference.py, the first joint location (which corresponds to the hip) is always [0,0,0] because we are choosing to ignore that joint in stats['dim_ignore_3d']. Is there a way to avoid setting the hip to (0,0,0)?

In my use case, I would like to avoid "fixing" or "pinning" a joint to the origin (or any other arbitrary point) for rendering purposes.

arrow keyboard doesn't work (Fixed: for matplotlib > 3)

Hi,

Great work!

When I try the 3D Skeleton Interactive Annotation using the pre-trained weights, the keyboard is not working correctly. 0-9 are fine, the figure shows the different selected limb when I click. However, arrow keyboards are not working:

Traceback (most recent call last):
  File "/anaconda3/envs/pytorch/lib/python3.6/site-packages/matplotlib/cbook/__init__.py", line 224, in process
    func(*args, **kwargs)
  File "annotate_3D.py", line 96, in press
    update_skeleton(angle_idx, event.key)
  File "annotate_3D.py", line 337, in update_skeleton
    update_global(angle_idx)
  File "annotate_3D.py", line 282, in update_global
    update_line(line_idx, parent_idx, child_idx)
  File "annotate_3D.py", line 265, in update_line
    lines[line_idx][0].set_3d_properties(z)
  File "/anaconda3/envs/pytorch/lib/python3.6/site-packages/mpl_toolkits/mplot3d/art3d.py", line 143, in set_3d_properties
    zs = np.broadcast_to(zs, xs.shape)
AttributeError: 'list' object has no attribute 'shape'

up, down, right or left all shows the same error. I tried debug thru printing the (event.key) and it shows that the received keyboard is correct (i.e. printing up, down, left, right when I click the corresponding key).

Could you please take a look on this? Thanks a lot!

about training data

The paper said there are 8M 2d-3d pairs, but the total number of the released h36m training data pairs is about 390k before augmentation, is that the correct number?

So if I guess right, that means you only take one h36m camera for each action instead of using 4 cameras for each action.

I don't know whether my guess is correct, so I post this issue to make sure.

Thank you in advance.

AttributeError: module 'libs.dataset.h36m' has no attribute 'data_utils'

Hi

Thx for this model. but i always get this error. Do you know whats the problem?

Test with a single image

Hi, is it possible to perform inference from a single image chosen by me? I've noticed that, inside the examples folder, there is a file called "inference.py" but it uses the file "example_annot.npy" that contains the pre-calculated 2D key-points of the test images you say to download. Will a version of the inference code be released that will allow a single image (thus different from the ones you propose) to be used to test the network?

Thanks

2D to 3D with own data

hi
I tried your 'inference.py' module with own 2d data, but the result has some problems：


The reason is that your output need to use human3.6's 3d_mean and std to decode, but my data will produce errors if I use this decoding method, So I want to ask you how can I solve this kind of problem.
thank you very much！

nothing

2d poses from 3d pose

hi
how can I extract 2d poses from the resulted 3d pose (2d pose front view, back view, and side view)
thanks

evolved dataset projection issue

Hi, I used your evolved program to evolved my own dataset with the default parameters, my own datset has 161 samples, and the evolved result is 419, but after I project the evolved result(419) to 2D and check its xy position, the valid(if min(xy) <0, invalid) 2D pose is only 47.

My dataset(world coordinate) is not in the meter unit like Human36M, does this affect my evolved result?

My dataset only has 17 joints, the other joints I set the value with 1, does this affect my evolved result?

In my understanding you only check the valid of 3D pose(world coordinate), and use all the evolved data to train, can this evolved strategy guarantee your 2D projection is valid?

Thank you very much

Question about the initial population

Hello. I want to ask some questions about initial 3d body-parts
I saw that the original 3d poses are loaded from the file "threeDPose_train.npy", and each sample has 96 dimensions.

I wonder what's the exact physical meaning of these values? Are they the absolute distance (use mm as the unit) in the local coordinate, where the center is the pelvis?

Running annotation_2D in py3 and fit_3d in py2

Hi,

As far as I understand, we first run annotation_2D.py in py3 environment, get the annotation.npy. Then we run the fit_3d.py in py2 environment to get the fitting parameters using smpl tool. However, fit_3d.py under py2 cannot load the annotation.npy since it is generated in py3 environment from annotation_2D.py. The error is shown as followed:

File "/anaconda3/envs/py2/lib/python2.7/site-packages/numpy/lib/format.py", line 701, in read_array
  array = pickle.load(fp, **pickle_kwargs)

ValueError: unsupported pickle protocol: 3

Am I missing something? Thanks in advance!

the input channel order to_spherical calculation

hello, thanks for your outstanding work. I am learning the evolution part and had some questions about the to_spherical calculation.

in EvoSkeleton/libs/skeleton/anglelimits.py line179-192 is the function of to_spherical(xyz), and I have one questions:

1. according to the code, before calling to_spherical(), the skeleton has been re_order to xzy, while the formula of to_spherical use the xyz order, so if the skeleton in xzy order input to to_spherical, can we still get the right result?

projection from 3d pose to 2d

Hello. Thanks for your excellent work.
I wonder after you obtain the 3d keypoint during evolution, do you randomly assume the camera location, and project the 3d keypoint to target the cameras? Does your project have the demo example?

Input of the plot_distribution in anglelimits.py

Hi, how can I obtain the H_temp?

AttributeError: module 'libs.utils' has no attribute 'utils'

Im training the network with 2Dto3Dnet.py with the cmd:
python 2Dto3Dnet.py -train True -num_stages 2 -ws True -ws_name "S1" -twoD_source "HRN/synthetic" -evolved_path "YourDataPath"
How ever it show the error:
Traceback (most recent call last):
File "2Dto3Dnet.py", line 16, in
import libs.utils.utils as utils
File "../libs/utils/utils.py", line 4, in
import libs.dataset.h36m.data_utils as data_utils
File "../libs/dataset/h36m/data_utils.py", line 15, in
import libs.utils.utils as utils
AttributeError: module 'libs.utils' has no attribute 'utils'
Anyone can help me
Many thanks

About the coordinate system

Hi. I want to consult something about the coordinate system.
I saw that before evolution, there is a function called to_local(), which "represents the bone vectors in the local coordinate systems"

EvoSkeleton/libs/skeleton/anglelimits.py

Line 285 in f31e7c2

def to_local(skeleton):

I feel confused about this part. Will the coordinate be converted from camera coordinate to local coordinate? And what's definition of local coordinate and what's the meaning of the conversion?
BTW, it will be my honour if you are willing to provide me with some references.
I am looking forward ton your reply.

about jointAngle file

not upload jointAngleModel_v2 file

ImportError: No module named libs.hhr.config

I'm getting this error when I run python ./examples/h36m2Dpose.py

Is all of the mpi data used?

happy spring festival,

I wonder about the details when you evaluate the model on the MPI-inf-3DHP dataset.

did you use all of the 8 subjects && 2 sequences && 14 cameras to inference the model?

Inconsistency between pretrained HRNet 2D detector and twoDPose_HRN.npy

Hi, appreciate for your excellent work and comprehensively technical detail release! I believe this would be great effort to 3DHPE field.

One question. I encountered some issues when inference HRNet model (I mean the 2D detector) that loads the pretrained weight, given cropped h36m images.

1. Accuracy in the first screenshot shows that the 2D average error (pixel) is around 7, which is inconsistent with reported 4.4.

2. Meanwhile, I print the 2D pose prediction for the frame of (9, 'Directions', 'Directions 1.54138969.h5-sh') from the model inference result, and from released twoDPose_HRN_test.npy.
   The inconsistence appears again as shown in the uploaded 2nd image.

Could you help me get rid of the unexpected situation ?
Did I miss something, or may you release another high-acc pretrained HR model ?

Many thanks !

Cannot use the annotation tool

After open the 2D annotation tool, the picture will show but cannot be labelled. Only the shortcut command for quit will work, the others won't. The 3D annotation cannot initiate for the missing file "jointAngleModel_v2.npy". Will you please re-upload the full package of this annotation tool, or did I ignore something?

About one of the pre-trained models

Hello,

Thank you for sharing this great work, very solid and helpful!

I would like to ask about the pre-trained model provided under 'Inference Example' in: https://github.com/Nicholasli1995/EvoSkeleton/blob/master/docs/TRAINING.md. The G-drive link is: https://drive.google.com/file/d/158oCTK-9Y8Bl9qxidoHcXfqfeeA7qT93/view

May I know which data do you use to train this model, as well as the parameters? I find it works perfectly in your proposed U3DPW dataset, while an EvoSkeleton model that I trained using synthetic H3.6m can only work on some of the example. Therefore, I wonder if this is because of the training data.

Many thanks,
Shiyang

What is S15678

Thanks for your nice work, the paper is a nice read.

I do not understand this: "We use S15678 as our initial population". What is S15678?