chunyuwang / imu-human-pose-pytorch

This is an official Pytorch implementation of "Fusing Wearable IMUs with Multi-View Images for Human Pose Estimation: A Geometric Approach, CVPR 2020".

Fusing Wearable IMUs with Multi-View Images for Human Pose Estimation: A Geometric Approach

Paper,

Installation

1. Clone this repo, and we'll call the directory that you cloned as ${POSE_ROOT}

2. Install dependencies.

3. Download pytorch imagenet pretrained models. Please download them under ${POSE_ROOT}/models, and make them look like this:

   ${POSE_ROOT}/models
   └── pytorch
       └── imagenet
           ├── resnet152-b121ed2d.pth
           ├── resnet50-19c8e357.pth
           └── mobilenet_v2.pth.tar
   

   They can be downloaded from the following link: Pretrained Model Download

Data preparation

For TotalCapture dataset, please download from official site and follow zhezh/TotalCapture-Toolbox to process data.

We have no permission to redistribute this dataset. Please do not ask us for a copy.

For precalculated pictorial model pairwise term, please download from HERE, and save in data/pict.

To reproduce our results in the paper, please download the trained models from HERE.

Testing

Testing ORN

python run/pose2d/valid.py \
--cfg experiments-local/totalcapture/res50_256_orn.yaml \
--model-file <path-to-your-download>/res50_256_final.pth.tar \
--gpus 0 --workers 1 \
--dataDir . --logDir log --modelDir output 

Testing ORPSM

python run/pose3d/estimate.py \
--cfg experiments-local/totalcapture/res50_256_orn.yaml \
--withIMU 1 \
--dataDir . --logDir log --modelDir output

Testing Baseline (SN + PSM)

python run/pose2d/valid.py \
--cfg experiments-local/totalcapture/res50_256_nofusion.yaml \
--model-file <path-to-your-download>/res50_256_final.pth.tar \
--gpus 0 --workers 1 \
--dataDir . --logDir log --modelDir output 

Then,

python run/pose3d/estimate.py \
--cfg experiments-local/totalcapture/res50_256_nofusion.yaml \
--withIMU 0 \
--dataDir . --logDir log --modelDir output

Training

Since our ORN and ORPSM has no learnable parameters, it can be conveniently appended to any 2D pose estimator. Thus training the SN backbone is sufficient.

python run/pose2d/train.py \
--cfg experiments-local/totalcapture/res50_256_nofusion.yaml \
--gpus 0 --workers 1 \
--dataDir . --logDir log --modelDir output

Citation

@inproceedings{zhe2020fusingimu,
  title={Fusing Wearable IMUs with Multi-View Images for Human Pose Estimation: A Geometric Approach},
  author={Zhang, Zhe and Wang, Chunyu and Qin, Wenhu and Zeng, Wenjun},
  booktitle = {CVPR},
  year={2020}
}

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

imu-human-pose-pytorch's Issues

Testing ORPSM

It's the fact that the data set is wrong and the first fifth is right, and the last four fifths are wrong
estimate3d_with_imu_res50_256_orn2023-10-17-06-08.csv

quickly_load_cam_1357.pkl

Could u upload the .pkl file for the camera parameters of No 1, 3, 5, 7. Thanks very much!

"outputs_catts_cat" is not found

Hi,

I got the error as NameError: name 'outputs_catts_cat' is not defined

'outputs_catts_cat' is not defined

Hi,
I got mistakes as 'outputs_catts_cat' is not defined

   estimate.py :    for idx_datum, prediction in enumerate(**outputs_catts_cat**)

MobileNetV2 cannot require

hello，could u upload the MobileNetV2.pth.tar in the onedrive again？I can not find it in the diretory that u provide for

NameError: name 'heatmaps' is not defined

hello,
When I use the orn configuration file to train, the code prompts "NameError: name 'heatmaps' is not defined".

KeyError: "There is no item named 'images/s_02_act_02_subact_01_ca_01/002485.jpg' in the archive"

Hi Chunyu@CHUNYUWANG：
According to the README document, I process the TotalCapture dataset using the TotalCapture-Toolbox and compress it into a zip file. I place the images.zip in the path: data/totalcapture/ and I find I can test smoothly. However, when I want to train with the instructions given，the following questions are prompted：
no checkpoint found at output/totalcapture/multiview_pose_resnet_50/res50_256_nofusion/checkpoint.pth.tar
KeyError: Caught KeyError in DataLoader worker process 0.
KeyError: "There is no item named 'images/s_02_act_02_subact_01_ca_01/002485.jpg' in the archive"
Do you know what's the problem? I will be appreciated if you can reply.

3D visualization

Hi @CHUNYUWANG You did a great work. I want to confirm that this repository contains code for 3D visualization or just it is for validation or testing?

which two coordinate systems are represented by R_{Ti}

Hello, I'm using your imu-human-pose-pytorch project and I'm hoping to input measurement data myself. However, I'm encountering issues with rotation matrices when doing so.

I'd like to ask, in the TotalCapture-Toolbox project, which two coordinate systems are represented by R_{Ti}?
From the section (3.4) implementation details of the paper "Real-Time Full-Body Motion Capture from Video and IMUs" referenced by the authors of the TotalCapture dataset, we understand that R_{ig} is assumed to be consistent between IMUs and aligned with the world coordinates through the global up direction and magnetic north.
Despite this, I'm still unable to understand which two coordinate systems RTi represents the transformation matrix for. Therefore, I'd like to ask for clarification from everyone. Thank you!

RuntimeError: expected scalar type Double but found Float

Hi，I‘m sorry to disturb you.I meet a problem when running your program.The details are as follows：
File "/root/Source/imu-human-pose/run/pose2d/../../lib/models/orn.py", line 86, in batch_uv_to_global_with_multi_depth coords_global = torch.bmm(inv_cam_extri_R, coords_img_frame_all_depth) + inv_cam_extri_T RuntimeError: expected scalar type Double but found Float
I try to change the data type as inv_cam_extri_R.to(torch.float64).It doesn't work.So I want to know if there is a solution.

Sorry to disturb you, I am in trouble while running your program

when I Testing ORN,I meet an error message"There is no item named 'images\\s_01_act_02_subact_02_ca_01\\000000.jpg' in the archive".

no such directory about quickload

I got the error as follows，
FileNotFoundError: [Errno 2] No such file or directory: '/data6/liwh/TotalCapture/quickload/totalcapture_quickload_validation_cam_1357.pkl'

How can I generate the quickload/xxxx.pkl ?