shaohua-pan / robustcap Goto Github PK

Code for our SIGGRAPH ASIA 2023 paper "Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture".

Home Page: https://shaohua-pan.github.io/robustcap-page/

License: MIT License

Python 100.00%

pose-estimation human-pose-estimation motion-capture

robustcap's Introduction

RobustCap

Code for our SIGGRAPH ASIA 2023 paper "Fusing Monocular Images and Sparse IMU Signals for Real-time Human

Motion Capture". This repository contains the system implementation and evaluation. See Project Page.

Installation

conda create -n RobustCap python=3.8
conda activate RobustCap
pip install -r requirements.txt

Install pytorch cuda version from the official website.

Data

SMPL Files, Pretrained Model and Test Data

Download smpl files from here or the official website. Unzip it and place it at models/.
Download the pretrained model and data and place them at data/.
For AIST++ evaluation, download the no aligned files and place it at data/dataset_work/AIST.

Evaluation

We provide the evaluation code for AIST++, TotalCapture, 3DPW and 3DPW-OCC. The results maybe slightly different from the numbers reported in the paper due to the randomness of the optimization.

python evaluate.py

Visualization

Visualization by open3d or overlay

We provide the visualization code for AIST++. You can use view_aist function in evaluate.py to visualize the results. By indicating seq_idx and cam_idx, you can visualize the results of a specific sequence and camera. Set vis=True to visualize the overlay results (you need to download the origin AIST++ videos and put them onto config.paths.aist_raw_dir). Use body_model.view_motion to visualize the open3d results.

Visualization by unity

You can use view_aist_unity function in evaluate.py to visualize the results. By indicating seq_idx and cam_idx, you can visualize the results of a specific sequence and camera.

Download unity assets from here.
Create a unity 3D project and use the downloaded assets, and create a directory UserData/Motion.
For the unity scripts, use Set Motion (set Fps to 60) and do not use Record Video.
Run view_aist_unity and copy the generated files to UserData/Motion.

Then you can run the unity scripts to visualize the results.

Live Demo

We use 6 Xsens Dot IMUs and a monocular webcam. For different hardwares, you may need to modify the code.

Config the IMU and camera parameters in config.Live.
Calibrate the camera. We give a simple calibration code in articulate/utils/executables/RGB_camera_calibration.py. Then copy the camera intrinsic parameters to config.Live.camera_intrinsic.
Connect the IMUs using the code articulate/utils/executables/xsens_dot_server_no_gui.py. Following the instructions in the command line including “connect, start streaming, reset heading, print sensor angle (make sure the angles are similar when you align the IMUs)”.
Run the live detector code live_detector.py and you can see the camera reading.
Run the Unity scene to render the results. You can write your own code or use the scene from Transpose (https://github.com/Xinyu-Yi/TransPose).
Run the live server code live_server.py to run our networks and send the results to Unity.

After doing this, you can see the real-time results in Unity. If you are encountering any problems, please feel free to issue.

Training

run net/sig_mp.py.

Citation

@inproceedings{pan2023fusing,
title={Fusing Monocular Images and Sparse IMU Signals for Real-time Human Motion Capture},
author={Pan, Shaohua and Ma, Qi and Yi, Xinyu and Hu, Weifeng and Wang, Xiong and Zhou, Xingkang and Li, Jijunnan and Xu, Feng},
booktitle={SIGGRAPH Asia 2023 Conference Papers},
pages={1--11},
year={2023}
}

robustcap's People

Contributors

Stargazers

Watchers

Forkers

jackzhousz stayyouth1993 saulocatharino point-1 seraph1188v 2132660698 seamoon76 lo4ding00 windwing2016

robustcap's Issues

是否可以分享demo测试用的unity工程啊？

Is there a training part to this code

If I want to train my own network, in which file should I make changes

optimize reprojection error

if self.use_reproj_opt and c > self.conf_range[0]:
p = j2dc[:, 2]#获取置信度
# optimize x, y
ax = (p / j[:, 2].pow(2)).sum() + self.smooth
bx = (p * (- j[:, 0] / j[:, 2].pow(2) + j2dc[:, 0] / j[:, 2])).sum()
ay = (p / j[:, 2].pow(2)).sum() + self.smooth
by = (p * (- j[:, 1] / j[:, 2].pow(2) + j2dc[:, 1] / j[:, 2])).sum()
d_tran = torch.tensor([bx / ax, by / ay, 0]).to(device)
tran = tran + d_tran
# optimize z
j = j + d_tran#这里已经更新了bx和by
az = (p * (j[:, 0].pow(2) + j[:, 1].pow(2)) / j[:, 2].pow(4)).sum() + self.smooth
bz = (p * ((j[:, 0] / j[:, 2] - j2dc[:, 0]) * j[:, 0] / j[:, 2].pow(2) + (
j[:, 1] / j[:, 2] - j2dc[:, 1]) * j[:, 1] / j[:, 2].pow(2))).sum()
d_tran = torch.tensor([0, 0, bz / az]).to(device)
请问在上述代码中az和bz的计算依据是什么

我的式子是正确的嘛，我无法推导出az和bz

重力常量计算

请问self.gravityc是如何得到的

请问一下传感器的方案开源吗？

How to enter a video

hi, I want to input video path, output a result video. What should I do？ Thank you

请问imus_addr这个参数

请问这个参数代表什么意思呢？指的imu设备地址吗？还是?还有Imu数据格式能否开源？

Can I have a image or video dataset?

Hi, I would like to access a dataset with IMU and video or images. So, can I have a raw image or video data? Thank you.

Questions about the acceleration

Hi,

Thanks for sharing the code of the awesome project!

I have some questions about the acceleration from the IMUs.

First, what is the unit of the acceleration? Is it SI (m/s^2) or G (g)?

Second, is the gravity cancelled out from the acceleration? I suppose the synthetic acc generated from the AIST++ dataset does not consider the gravity, but the readings from IMUs contain the gravity.

Thanks.