bearpaw / pyranet

Code for "Learning Feature Pyramids for Human Pose Estimation" (ICCV 2017)

Learning Feature Pyramids for Human Pose Estimation

Training and testing code for the paper

Learning Feature Pyramids for Human Pose Estimation
Wei Yang, Shuang Li, Wanli Ouyang, Hongsheng Li, Xiaogang Wang
ICCV, 2017

This code is based on stacked hourglass networks and fb.resnet.torch. Thanks to the authors.

Install

1. Install Torch.

2. Install dependencies.

   luarocks install hdf5
   luarocks install matio
   luarocks install optnet
   

3. (Optional) Install nccl for better performance when training with multi-GPUs.

   git clone https://github.com/NVIDIA/nccl.git
   cd nccl
   make 
   make install
   luarocks install nccl
   

   set LD_LIBRARY_PATH in file ~/.bashrc if libnccl.so is not found.

4. Prepare dataset.
   Create a symbolic link to the images directory of the MPII dataset:

   ln -s PATH_TO_MPII_IMAGES_DIR data/mpii/images
   

   Create a symbolic link to the images directory of the LSP dataset (images are stored in PATH_TO_LSP_DIR/images):

   ln -s PATH_TO_LSP_DIR data/lsp/lsp_dataset
   

   Create a symbolic link to the images directory of the LSP extension dataset (images are stored in PATH_TO_LSPEXT_DIR/images):

   ln -s PATH_TO_LSPEXT_DIR data/lsp/lspet_dataset
   

Training and Testing

Quick Start

Testing from our pretrained model

Download our pretrained model to ./pretrained folder from Google Drive. Test on the MPII validation set by running the following command

qlua main.lua -batchSize 1 -nGPU 1 -nStack 8 -minusMean true -loadModel pretrained/model_250.t7 -testOnly true -debug true

For multi-scale testing, run

qlua evalPyra.lua -batchSize 1 -nGPU 1 -nStack 8 -minusMean true -loadModel pretrained/model_250.t7 -testOnly true -debug true

Note:

• If you DO NOT want to visualize the training results. Set -debug false and use th instead of qlua.
• you may set the number of scales in evalPyra.lua (Line 22 ). Use fewer number of scales or multiple GPUs if "out of memory" occurs.
• use -loadModel MODEL_PATH to load a specific model for testing or training

Train a two-stack hourglass model

Train an example two-stack hourglass model on the MPII dataset with the proposed Pyramids Residual Modules (PRMs)

sh ./experiments/mpii/hg-prm-stack2.sh 

Customize your own training and testing procedure

A sample script for training on the MPII dataset with 8-stack hourglass model.

#!/usr/bin/env sh
expID=mpii/mpii_hg8   # snapshots and log file will save in checkpoints/$expID
dataset=mpii          # mpii | mpii-lsp | lsp |
gpuID=0,1             # GPUs visible to program
nGPU=2                # how many GPUs will be used to train the model
batchSize=16          
LR=6.7e-4
netType=hg-prm        # network architecture
nStack=2
nResidual=1
nThreads=4            # how many threads will be used to load data
minusMean=true
nClasses=16
nEpochs=200           
snapshot=10           # save models for every $snapshot

OMP_NUM_THREADS=1 CUDA_VISIBLE_DEVICES=$gpuID th main.lua \
   -dataset $dataset \
   -expID $expID \
   -batchSize $batchSize \
   -nGPU $nGPU \
   -LR $LR \
   -momentum 0.0 \
   -weightDecay 0.0 \
   -netType $netType \
   -nStack $nStack \
   -nResidual $nResidual \
   -nThreads $nThreads \
   -minusMean $minusMean \
   -nClasses $nClasses \
   -nEpochs $nEpochs \
   -snapshot $snapshot \
   # -resume checkpoints/$expID  \  # uncomment this line to resume training
   # -testOnly true \               # uncomment this line to test on validation data
   # -testRelease true \            # uncomment this line to test on test data (MPII dataset)

Evaluation

You may evaluate the PCKh score of your model on the MPII validation set. To get start, download our prediction pred_multiscale_250.h5 to ./pretrained from Google Drive, and run the MATLAB script evaluation/eval_PCKh.m. You'll get the following results

      Head , Shoulder , Elbow , Wrist , Hip , Knee  , Ankle , Mean , 
name , 97.41 , 96.16 , 91.10 , 86.88 , 90.05 , 86.00 , 83.89 , 90.27

Citation

If you find this code useful in your research, please consider citing:

@inproceedings{yang2017pyramid,
    Title = {Learning Feature Pyramids for Human Pose Estimation},
    Author = {Yang, Wei and Li, Shuang and Ouyang, Wanli and Li, Hongsheng and Wang, Xiaogang},
    Booktitle = {arXiv preprint arXiv:1708.01101},
    Year = {2017}
}

pyranet's Issues

high acc in train and very low acc in test ,i use the prms in my pytorch code .

First, I really appreciate your work！
I want to write prms in pytorch.But the problem is ,loss and acc (badly)basically unchanged in the test set.Where did I go wrong?
thanks

How can get my hdf5 annotation data?

I want to train the model from a different dataset, then I find a problem, how can I get the hdf5 annotation data by myself? Could the author publish the code about how to transform the label data in the original dataset to the hdf5 formation?

where is the parameter initialization code described in your paper?

Hi,
I haven't found the parameter initialization described in your paper in this implementation.

Obtaining skeleton coordinates

Hi!

Thank you for publishing your work!

I was looking through the code for a way to visualize detections as skeletal coordinates. I was indeed able to find a drawSkeleton() function, which, in itself is being called by drawOuptput() and drawFeature(). Yet, in all calls to drawOutput() the "coords" argument is omitted, while there are no calls to drawFeature() at all. The following condition (in drawOutput()) is thus never satisfied:

if coords then
    im = drawSkeleton(input, coords, hms)
end

Am I missing something?

Thanks in advance!

Where can I download the extended lip data set?

Hi, Thanks for your code!
But I just found original LSP dataset with 2000 images in the official website.
I am very glad for your guidance!

How to generate 'scale' and 'objpos'?

Hi, Thanks for your code!
The 'scale' and 'objpos' is also not provided offictially. How do you generate them?
Expect for your guidance! Thank you!

make predictions on MPII testing set

Hi,

Thanks so much for your sharing the codes.

How can I use a trained model to make predictions on the MPII testing set and generate the .MAT file that can be evaluated by the mpii website http://human-pose.mpi-inf.mpg.de? Have you implemented such functions in your codes?

Can anyone share the LSP dataset

the official link of LSP dataset is broken, can anyone share the LSP dataset. Thanks a lot

PyTorch Version?

Thanks for your great work!
Wonder that whether you would update it to PyTorch in your short-term plan? Thanks.

How to get the [email protected] on MPII test set, and [email protected] on Lip test set

May I ask that the prediction of single person attitude estimation is, the first step command is not quite right, how to operate

annolist_test = annolist(RELEASE.img_train == 0);

When I type an original command，it occurs an error: Undefined function or variable 'Imgidx' . How to solve ig
How should the command on lines 2-4 proceed?Where should I find the assignment of three variables?

Can you share the code about LSP dataset of Pyranet(lua)?

I have a SCI overhaul needs the result of [email protected]. Thank you very much

fine tune your model

Hi, bearpaw
I am using your code to train my own dataset. Is it possible to fine tune on your model? My dataset has different keypoint from yours. Thanks.

How 头

Do you just use 'LEEDS_annotations.json' to train a model for LSP?

Sorry for multiple open issue. I just hesitate how to express my doubts and I am newcomer to github.
I am not skilled at pytorch. Therefore, I am not sure if your code uses 'LEEDS_annotations.json' directly to train a model on LSP dataset.
Thank you very much!

frame rate

@bearpaw What about the frame rate and the hardware specification of this implementation ?

How to train a model for LSP dataset?

Thank you for sharing your wonderful work!
Can I just use 'LEEDS_annotations.json' to train a model for LSP?
Thank you!

Where is the "c[2] = c[2] + 15 * s s = s * 1.25" for the MPII?

Hi,

Thanks for sharing your code!
I remember in the original hourglass code there are
`-- Small adjustment so cropping is less likely to take feet out

c[2] = c[2] + 15 * s
s = s * 1.25

`
before the crop operation.
However, I can't find the corresponding code in your project. It seems you only multiply a scale factor in the multi-scale testing. In the training and single scale testing, you only use the provided scale and center without any adjustment. Do I understand right? Do you think it's unnecessary for the above adjustment?

Thanks

Realtime

How is the realtime perfommance? It's said that this model needs 45.9GFLOPs per image. Does it works on videos?

Your Final model

Thank you for sharing your wonderful work!

Is the model in Google Drive the same one you used to get 1st on MPII ? If that's so, that means you finally choose a 8-stack structure?

Have you done some experiments about number of stack based on your RPM ? For example, is there a big gap on performance between 2-stack and 8-stack?

Looking forward to your response!

Aboat six-pyramid test problem?

When testing the six pyramids, you need to first change the 64 * 64 size of the heat map into 256 * 256, and then change it into the original size, such as 720 * 1280. If so, why do you need to go through 256 * 256, and in the process of transformation, does the center need to change according to the same scaling factor?

Evaluation on LSP Dataset

Hi,@bearpaw
Thank you for sharing your wonderful work!
Based on the steps you provided, after training on the LSP data set, generate a .h5 prediction file, but in this evaluation code: http://human-pose.mpi-inf.mpg.de/results/lsp/evalLSP. Zip, it needs to use .mat files to make predictions. So what I want to ask is how to generate such a .mat file ? Have you implemented such functions in your codes?

confusing code

local ResidualUp = n >= 2 and ResidualPyramid or Residual

as i am not familar with lua, what does this mean ? if n >=2 use res_pyra and others residual? but the parameters of the two function is not the same