garvita-tiwari / leap

[CVPR'21] LEAP: Learning Articulated Occupancy of People https://neuralbodies.github.io/LEAP

LEAP: Learning Articulated Occupancy of People

Paper | Video | Project Page

This is the official implementation of the CVPR 2021 submission LEAP: Learning Articulated Occupancy of People

LEAP is a neural network architecture for representing volumetric animatable human bodies. It follows traditional human body modeling techniques and leverages a statistical human prior to generalize to unseen humans.

If you find our code or paper useful, please consider citing:

@InProceedings{LEAP:CVPR:21,
  title = {{LEAP}: Learning Articulated Occupancy of People},
  author = {Mihajlovic, Marko and Zhang, Yan and Black, Michael J and Tang, Siyu},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2021},
}

Contact Marko Mihajlovic for questions or open an issue / a pull request.

Prerequests

1) SMPL body model

Download a SMPL body model (SMPL, SMPL+H, SMPL+X, MANO) and store it under ${BODY_MODELS} directory of the following structure:

${BODY_MODELS}
├── smpl
│   └── x
├── smplh
│   ├── male
|   │   └── model.npz
│   ├── female
|   │   └── model.npz
│   └── neutral
|       └── model.npz
├── mano
|   └── x
└── smplx
    └── x

NOTE: currently only SMPL+H model is supported. Other models will be available soon.

2) Installation

Another prerequest is to install python packages specified in the requirements.txt file, which can be conveniently accomplished by using an Anaconda environment:

# clone the repo
git clone https://github.com/neuralbodies/leap.git
cd ./leap

# create environment
conda env create -f environment.yml
conda activate leap

and install the leap package via pip:

# note: install the build-essentials package if not already installed (`sudo apt install build-essential`) 
python setup.py build_ext --inplace
pip install -e .

3) (Optional) Download LEAP pretrained models

Download LEAP pretrained models from here and extract them under ${LEAP_MODELS} directory.

Usage

Check demo code in examples/query_leap.py for a demonstration on how to use LEAP for differentiable occupancy checks.

Train your own model

Follow instructions specified in data_preparation/README.md on how to prepare training data. Then, replace placeholders for pre-defined path variables in configuration files (configurations/*.yml) and execute training_code/train_leap.py script to train the neural network modules.

LEAP consists of two LBS networks and one occupancy decoder.

cd training_code

To train the forward LBS network, execute the following command:

python train_leap.py ../configurations/fwd_lbs.yml

To train the inverse LBS network:

python train_leap.py ../configurations/inv_lbs.yml

Once the LBS networks are trained, execute the following command to train the occupancy network:

python train_leap.py ../configurations/leap_model.yml

See specified yml configuration files for details about network hyperparameters.