dwangomediavillage / 3dpose_gan

The authors' implementation of Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locations

Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locations

This is the authors' implementation of Unsupervised Adversarial Learning of 3D Human Pose from 2D Joint Locations

Run Inference for demo (with openpose)

1. Download openpose pretrained model
   • openpose_pose_coco.prototxt
     • https://github.com/opencv/opencv_extra/blob/3.4.1/testdata/dnn/openpose_pose_coco.prototxt
   • pose_iter_440000.caffemodel
     • http://posefs1.perception.cs.cmu.edu/OpenPose/models/pose/coco/pose_iter_440000.caffemodel
2. Run Inference
   • python bin/demo.py sample/image.png --lift_model sample/gen_epoch_500.npz --model2d pose_iter_440000.caffemodel --proto2d openpose_pose_coco.prototxt
   • Need OpenCV >= 3.4
     • < 3.3 results extreamly wrong estimation

Dependencies(Recommended versions)

• Python 3.6.5
• Cupy 4.0.0
• Chainer 4.0.0
• OpenCV 3.4 (when showing results)
• git-lfs
  • to download pre-trained model
  • or you can download pre-trained model directory from https://github.com/DwangoMediaVillage/3dpose_gan/blob/master/sample/gen_epoch_500.npz?raw=true

Training

Human3.6M dataset

• Unsupervised learning of 3D points from ground truth 2D points

  python bin/train.py --gpu 0 --mode unsupervised --dataset h36m --use_heuristic_loss
  

• Unsupervised learning of 3D points from detected 2D points by Stacked Hourglass

  TBA

• Supervised learning of 3D points from ground truth 2D points

  python bin/train.py --gpu 0 --mode supervised --activate_func relu --use_bn
  

MPII dataset

TBA

MPI-INF-3DHP dataset

TBA

Evaluation

python bin/eval.py results/hoge/gen_epoch_*.npz

3dpose_gan's Issues

Pose Translation is not incorporated?

So if the character moves along an axis, he is still zero centered in animation. How can I add translation along axes?

Thanks

How to get gen_epoch_500.npz for custom dataset?

I would like to train for my custom dataset (Not Human Pose).
So, how to get gen_epoch_500.npz for my dataset?

how to create hd5 data file for h36M dataset?

I would like to train my custom dataset.
So, how can I create hd5 data file for dataset and can I use gen_epoch_500.npz in my training?

Do you need 3D annotation points to train your GAN?

I read your paper in details.
So, for training you said you do not need 3D annotation set to train your GAN.
In here, I have an unclear point.
You trained Human3.6M dataset in your GAN without 3D annotation data. And then estimate z component. After your estimation, there are estimated 3D pose.
My question is how to compare your estimated 3D pose and ground truth 3D pose?
For comparison, do we need 3D annotation data? Because I have not 3D pose for my interest field (i.e not human pose).
To compare the experimental results, I think I will require 3D pose. Is it right?
Thank your for your reading.

upper body?

this can be used on upper body?

pixel coordinates or image plane coordinates??

Hi,Excuse me, I have a little problem about some details in your code.

In H3.6m dataset you used 3D data and real camera projection to generate 2D data, so the generated 2D points data are pixel coordinates instead of image plane coordinates because of multiplying the parameter f in the camera.

So my problem is whether I need to divide f when normalizing the 2D points in order to get the image plane coordinates.The reason for this is because the assumption is that the orthogonal projection, that is, the 2D coordinates generated by the generator, are the image plane coordinates,while the input are pixel coordinates, which I think it will increase the error.

Thanks

Dimension mismatch when train MPII dataset?

How to train using MPII only 2D dataset(mpii_poses.npy)?
Is it the same with H36M?
When I train using MPII dataset, I got this error:
Expect: x0.shape == x1.shape Actual: (16, 17) != (16, 0, 51)

cv2.error when using demo.py

When I run demo.py I met this error.Can you give me some advices to deal with it?
python3 bin/demo.py test.jpg --lift_model sample/gen_epoch_500.npz --model pose_iter_440000.caffemodel --proto2d openpose_pose_coco.prototxt

MODEL: generator, N_OUT: 17, N_UNIT: 1024
[mjpeg @ 0x55f4839aa660] mjpeg: invalid TIFF header in EXIF data
[mjpeg @ 0x55f4839aa220] mjpeg: invalid TIFF header in EXIF data
[ INFO:0] Initialize OpenCL runtime...
OpenCV Error: Assertion failed (output_slice.isContinuous() && output_slice.size == curr_output.size) in fuseLayers, file /io/opencv/modules/dnn/src/dnn.cpp, line 1430
Traceback (most recent call last):
File "bin/demo.py", line 201, in
main(args)
File "bin/demo.py", line 163, in main
points = OpenPose(args).predict(args, frame)
File "bin/demo.py", line 122, in predict
out = self.net.forward()
cv2.error: /io/opencv/modules/dnn/src/dnn.cpp:1430: error: (-215) output_slice.isContinuous() && output_slice.size == curr_output.size in function fuseLayers

Cannot download gen_epoch_500.npz

It seems the git lfs has a small quota, which has been exceeded. Could you provided another way to get the gen_epoch_500.npz?

batch response: This repository is over its data quota. Purchase more data packs to restore access. error: failed to fetch some objects from 'https://github.com/DwangoMediaVillage/3dpose_gan.git/info/lfs'

An error on no person case

When I tried to use /dev/video0 as input or no person image such as dog416.png of opencv_extra, I met an error such as follows. Is it correct response?

$ python bin/demo.py dog416.png --lift_model sample/gen_epoch_500.npz --model pose_iter_440000.caffemodel --proto2d openpose_pose_coco.prototxt
MODEL: generator, N_OUT: 17, N_UNIT: 1024
[ INFO:0] Initialize OpenCL runtime...
Traceback (most recent call last):
File "bin/demo.py", line 201, in
main(args)
File "bin/demo.py", line 167, in main
points = np.float32(to36M(points, BODY_PARTS))
File "bin/demo.py", line 64, in to36M
bones[body_parts['Nose']] - thorax
TypeError: unsupported operand type(s) for -: 'NoneType' and 'float'

Error for OSError: Failed to interpret file 'sample/gen_epoch_500.npz' as a pickle?

I got this error when I run your demo.py
"OSError: Failed to interpret file 'sample/gen_epoch_500.npz' as a pickle"
How to fix it?

Typo in running command

Thanks for sharing this work. When running the demo, I guess the right command is

python bin/demo.py sample/image.png --lift_model sample/gen_epoch_500.npz --model2d pose_iter_440000.caffemodel --proto2d openpose_pose_coco.prototxt

One argument is model2d instead of model.

Question: _pickle.UnpicklingError: invalid load key, 'v'

I tried to run the project on Ubuntu 16.04, but I met an error.
The environment is clean installed one.
Could you give me some advices on what I should do with this error?

----- logs ------
cvl@cvl-All-Series:~/workspace/3dpose_gan$ python bin/demo.py your/image.png --lift_model sample/gen_epoch_500.npz --model pose_iter_440000.caffemodel --proto2d pose_deploy_linevec.prototxt
MODEL: generator, N_OUT: 17, N_UNIT: 1024
Traceback (most recent call last):
File "/home/cvl/.pyenv/versions/3.6.1/lib/python3.6/site-packages/numpy/lib/npyio.py", line 428, in load
return pickle.load(fid, **pickle_kwargs)
_pickle.UnpicklingError: invalid load key, 'v'.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "bin/demo.py", line 201, in
main(args)
File "bin/demo.py", line 157, in main
chainer.serializers.load_npz(args.lift_model, model)
File "/home/cvl/.pyenv/versions/3.6.1/lib/python3.6/site-packages/chainer/serializers/npz.py", line 151, in load_npz
with numpy.load(file) as f:
File "/home/cvl/.pyenv/versions/3.6.1/lib/python3.6/site-packages/numpy/lib/npyio.py", line 431, in load
"Failed to interpret file %s as a pickle" % repr(file))
OSError: Failed to interpret file 'sample/gen_epoch_500.npz' as a pickle

cvl@cvl-All-Series:/workspace/3dpose_gan$ ls sample/gen_epoch_500.npz
sample/gen_epoch_500.npz
cvl@cvl-All-Series:/workspace/3dpose_gan$ pip install -U pickle
Collecting pickle
Could not find a version that satisfies the requirement pickle (from versions: )
No matching distribution found for pickle
cvl@cvl-All-Series:/workspace/3dpose_gan$ pip install -U scipy
Requirement already up-to-date: scipy in /home/cvl/.pyenv/versions/3.6.1/lib/python3.6/site-packages
Requirement already up-to-date: numpy>=1.8.2 in /home/cvl/.pyenv/versions/3.6.1/lib/python3.6/site-packages (from scipy)
cvl@cvl-All-Series:/workspace/3dpose_gan$ python -V
Python 3.6.1

cannot import name 'pose'

When "run" demo the following error will be output
Module versions are consistent.

Traceback (most recent call last):
File "bin/demo.py", line 9, in
import evaluation_util
File "/home/usr/download/3dpose_gan-master/bin/evaluation_util.py", line 10, in
import projection_gan
File "/usr/local/lib/python3.6/site-packages/projection_gan-0.0.1-py3.6.egg/projection_gan/init.py", line 1, in
ImportError: cannot import name 'pose'

Do you have any advice?

cannot download gen_epoch_500.npz

you purchased additional data pack, closed that issue.
but,it seems happen again.....

Extending 3dpose_gan to posenet ?

Hi is it possible to extend this to posenet (which does the 2d joint detection? ) It has the following keypoint configuration

'nose', 0 'leftEye', 1 'rightEye', 2 'leftEar', 3 'rightEar', 4 'leftShoulder', 5 'rightShoulder', 6 'leftElbow', 7 'rightElbow', 8 'leftWrist', 9 'rightWrist', 10 'leftHip', 11 'rightHip', 12 'leftKnee', ' 13 rightKnee', 14 'leftAnkle', 15 'rightAnkle' 16

cv2.error, ver problem?

When "run" demo the following error will be output
Module versions are consistent.

[libprotobuf ERROR /io/opencv/3rdparty/protobuf/src/google/protobuf/text_format.cc:288] Error parsing text-format opencv_caffe.NetParameter: 7:1: Expected identifier, got: <
Traceback (most recent call last):
  File "bin/demo.py", line 199, in <module>
    main(args)
  File "bin/demo.py", line 163, in main
    points = OpenPose(args).predict(args, frame)
  File "bin/demo.py", line 108, in __init__
    self.net = cv.dnn.readNetFromCaffe(args.proto2d, args.model2d)
cv2.error: OpenCV(3.4.5) /io/opencv/modules/dnn/src/caffe/caffe_io.cpp:1151: error: (-2:Unspecified error) FAILED: ReadProtoFromTextFile(param_file, param). Failed to parse NetParameter file: sample/openpose_pose_coco.prototxt in function 'ReadNetParamsFromTextFileOrDie'

Do you have any advice?