arthur151 / dynacam

[CVPR 2023] DynaCam dataset - 3D human trajectories in global coordinates from videos captured by dynamic cameras

DynaCam

| [Paper] | [Video] | [Project Page] |

DynaCam contains in-the-wild RGB videos captured by dynamic cameras, including annotations:

• 3D human trajectories in world coordinates

For the details, please refer to our project page.

Download

[Google drive]

[Baidu drive (百度网盘)]

The structure of dataset is supposed to be:

|-- DynaCam
| --|-- video_frames
|   |   |-- panorama_test
|   |   |-- panorama_train
|   |   |-- panorama_val
|   |   |-- translation_test
|   |   |-- translation_train
|   |   |-- translation_val
|   |-- annotations
|   |   |-- *.npz

Visualization

To visualize each video sequences and corresponding annotations, like 3D human trajectory, please download the SMPL_NEUTRAL.pkl and put it into 'assets/' , then run

sh install.sh
# set the path to dynacam_folder in show_examples.py 
python show_examples.py 

Press `stop` to stop the animation, draw the `slider` to sellect the frame, press `ESC` on your keyboard to go next.

Evaluation

To re-implement all results on DynaCam in our paper, please download predictions, set the path in evaluation.py to ensure the structure like

|-- predictions
| --|-- TRACE
| --|-- GLAMR
| --|-- bev_dpvo

, then run:

sh install.sh
python evaluation.py

Citation

Please cite our paper if you use DynaCam in your research.

@InProceedings{TRACE,
    author = {Sun, Yu and Bao, Qian and Liu, Wu and Mei, Tao and Black, Michael J.},
    title = {{TRACE: 5D Temporal Regression of Avatars with Dynamic Cameras in 3D Environments}}, 
    booktitle = {IEEE/CVF Conf.~on Computer Vision and Pattern Recognition (CVPR)}, 
    month = June, 
    year = {2023}}

dynacam's Issues

Dataset annotations

Thanks a lot for your interesting work. Is it possible to provide a description of the dataset annotations and how they were computed? What is the difference between panorama and translation frames?
Thank you in advance for your help.

Extracting pose data on a random test video

Hi, Thanks for the code!
Can you provide with the steps to execute in order to extract pose data on a random test video.

Issues with dataset annotations

Thank you for your excellent work! I was just wondering about some issues with the dataset -

1. Intrinsics not aligned with frame sizes, e.g., mpii-bicycling-bicycling,BMX-056019255-0 from translation_train has frame sizes of [[520 595], [525 595], [529 595], [532 595], [536 595], [537 595], [537 595], [538 595], [543 595], [551 595], [561 595], [572 595], [584 595], [595 582]], but the intrinsic annotations are with principle points (640, 360). This does not make sense.
2. The frame sizes differ for frames in a single video, with the same example as above.
3. SMPL annotations are only partial - some videos with humans in the whole video are not with SMPL annotations for all frames.
4. Not all sequences have annotations. For example, panorama_train has 76 sequences but only 47 are annotated.

Thanks in advance for your clarification. Looking forward to your reply.