otaheri / grip

GRIP: Generating Interaction Poses Using Spatial Cues and Latent Consistency

[Paper Page ] [ArXiv Paper ]

Video

Check out the YouTube video below for more details.

Table of Contents

• Description
• Installation
• Getting Started
• Examples
• Citation
• License
• Acknowledgments
• Contact

Description

This repository Contains:

• Code to preprocess and prepare the GRAB data for the GRIP paper
• Retraining GRIP models, allowing users to change details in the training configuration.
• Code for generating results on the test set split of the data
• Tools to visualize and save generated sequences from GRIP

Requirements

The models are built in PyTorch 1.7.1 and tested on Ubuntu 20.04 (Python3.8, CUDA11.0).

Some of the requirements are:

• Python ==3.8
• Pytorch==1.7.1
• SMPL-X
• bps_torch
• psbody-mesh

Installation

To install the requirements please follow the next steps:

• Clone this repository and install the requirements:

  git clone https://github.com/otaheri/GRIP

  cd GRIP
  pip install -r requirements.txt
  

If there are issues with installing the psbody-mesh package, please follow the instructions on the original repo to install it from the source.

Getting started

In order to use GRIP please follow the steps below:

• Download the GRAB dataset from our website and make sure to follow the steps there.
• Follow the instructions on the SMPL-X website to download SMPL-X models.
• Check the Examples below to process the required data, ude pretrained GRIP models, and to train GRIP models.

data

• Download the GRAB dataset from the GRAB website, and follow the instructions there to extract the files.
• Process the data by running the command below.

python data/process_data.py --grab-path /path/to/GRAB --smplx-path /path/to/smplx/models/ --out-path /the/output/path

CNet, RNet, and ANet models

• Please download these models from our website and put them in the folders as below.

    GRIP
    ├── snapshots
    │   │
    │   ├── anet.pt
    │   ├── cnet.pt
    │   └── rnet.pt
    │   
    │
    .
    .
    .

Examples

• Generate hand interaction motion for the test split - CNet and RNet.

  python train/infer_cnet.py --work-dir /path/to/work/dir --grab-path /path/to/GRAB --smplx-path /path/to/models/ --dataset-dir /path/to/processed/data

• Train CNet and RNet with new configurations

  To retrain these models with a new configuration, please use the following code.

  python train/train_cnet.py --work-dir path/to/work/dir --grab-path /path/to/GRAB --smplx-path /path/to/models/ --dataset-dir /path/to/processed/data --expr-id EXPERIMENT_ID
  

• Generate denoised arm motions and visualize them using ANet on the test split

  python train/infer_anet.py --work-dir path/to/work/dir --grab-path /path/to/GRAB --smplx-path /path/to/models/ --dataset-dir /path/to/processed/data

• Train ANet for denoising arms with new training configurations

  To retrain ANet with a new configuration, please use the following code.

  python train/train_anet.py --work-dir path/to/work/dir --grab-path /path/to/GRAB --smplx-path /path/to/models/ --dataset-dir /path/to/processed/data --expr-id EXPERIMENT_ID
  

Citation

@inproceedings{taheri2024grip,
  title  = {{GRIP}: Generating Interaction Poses Using Latent Consistency and Spatial Cues},
  author = {Omid Taheri and Yi Zhou and Dimitrios Tzionas and Yang Zhou and Duygu Ceylan and Soren Pirk and Michael J. Black},
  booktitle = {International Conference on 3D Vision ({3DV})},
  year = {2024},
  url = {https://grip.is.tue.mpg.de}
}

License

Software Copyright License for non-commercial scientific research purposes. Please read carefully the LICENSE page for the terms and conditions and any accompanying documentation before you download and/or use the GRIP data, model and software, (the "Data & Software"), including 3D meshes (body and objects), images, videos, textures, software, scripts, and animations. By downloading and/or using the Data & Software (including downloading, cloning, installing, and any other use of the corresponding github repository), you acknowledge that you have read and agreed to the LICENSE terms and conditions, understand them, and agree to be bound by them. If you do not agree with these terms and conditions, you must not download and/or use the Data & Software. Any infringement of the terms of this agreement will automatically terminate your rights under this LICENSE.

Acknowledgments

This work was partially supported by Adobe Research (during the first author's internship), the International Max Planck Research School for Intelligent Systems (IMPRS-IS), and the German Federal Ministry of Education and Research (BMBF): Tübingen AI Center, FKZ: 01IS18039B.

We thank:

• Tsvetelina Alexiadis and Alpár Cseke for the Mechanical Turk experiments.
• Benjamin Pellkofer for website design, IT, and web support.

Contact

The code of this repository was implemented by Omid Taheri.

For questions, please contact [email protected].

For commercial licensing (and all related questions for business applications), please contact [email protected].

grip's Issues

Release Date

Very cool work! Do you have any expected timeline of code release?

Wrist index in data processing-'add wrist perturbation' part

Thanks for the great work. I have some questions about the codes.

1. In data processing part, I found you add some perturbation to joint #18(right elbow) and #19(left wrist)

   GRIP/data/process_data.py

   Line 336 in 64b160e

   motion_sbj_p_arms['body_pose'][:,3*18:3*19] += to_tensor(aug_rw_p)

   But according to the paper and code comments, here should be left and right elbow. Same problem for adding shoulder perturbation in next few lines. I follow the index in Grab code here https://github.com/otaheri/GRAB/blob/284cba757bd10364fd38eb883c33d4490c4d98f5/tools/utils.py#L166
2. From my understanding of your code, the sbj_output_p is for RNet training and sbj_output_p_arms is for ANet(if wrong please correct me...). But here you send sbj_output_p_arms params to sbj_output_p. Is this a typo?

   GRIP/data/process_data.py

   Line 356 in 64b160e

   sbj_output_p = sbj_m_h(**motion_sbj_p_arms, return_full_pose=True)

Looking forward to your reply. Thank you.