Official PyTorch implementation of DeepNAG.
DeepNAG is a novel RNN-based sequence generator that can reliably create synthetic 2D/3D gestures. Instead of relying on generative adversarial networks (GAN) to train a sequence generator, DeepNAG uses a standalone loss function based on soft dynamic time warping (sDTW) and Hausdorff distance. Our novel loss function is intuitive, runs fast and yields great results. Please see our publication for more details.
DeepNAG's architecture is fairly simple, and only consists of gated recurrent units (GRU) and fully-connected layers:
This repository contains:
- DeepNAG's implementation
- DeepGAN's implementation (GAN-based gesture generation)
- Pretrained DeepNAG and DeepGAN models (found under
pretrained_models
)
The following is some sample synthetic gestures from the JK2017 (Kinect) dataset. In both animations, the black skeleton is an actual person while the remaining skeletons are synthetic results produced by DeepNAG!
The following is some sample synthetic gestures from the $1-GDS dataset (produced by DeepNAG). The red samples are human drawn, while the black samples are synthetic. The last two rows are the overlayed renderings of some randomly selected real and synthetic samples to demonstrate the diversity of the generated samples compared to the real ones.
The following is the list of requirements for this project:
- Python v3.6+
- PyTorch v1.4+
- Numpy (will be installed along PyTorch)
- Matplotlib (for visualization)
- pytorch-softdtw-cuda (included as a git submodule)
- JitGRU (included as a git submodule) (needed for DeepGAN only)
- Numba (preferably install via your OS's package manager)
- (Optional) Tensorboard to monitor the training process
- (Optional) CUDA toolkit with an NVIDIA GPU (for faster training, although CPU-only training still works, but is very slow)
All the requirements are included in requirements.txt
.
Training a model from scratch involves 3 easy steps:
- Obtain the sources and initialize git submodules
git clone https://github.com/Maghoumi/DeepNAG.git
cd DeepNAG
git submodule update --init --recursive
- Install the dependencies (make sure the correct
pip
is used)
pip install -r requirements.txt
- Run the code (make sure the correct
python
v3.6+ is used)
python main.py --model=DeepNAG --dataset=dollar-gds --use-tensorboard=1
The above code will download the $1-GDS dataset and train a DeepNAG generator on the entire dataset. The dataset will be downloaded to DeepNAG/data
and the run results will be dumped under DeepNAG/logs/unique-session-name
.
The training progress will be showed in the standard output. The training progress will additionally be written to tensorboard event files under DeepNAG/logs/unique-session-name/tensorboard
. By default, training will run for 25000 epochs, which is enough to get good results on the $1-GDS dataset.
The JK2017 (Kinect) dataset is included in this repository. To train a model on this dataset, run
python main.py --model=DeepNAG --dataset=jk2017-kinect
Training will need to run for at least 5000 epochs to produce good results. If you do not pass the --epoch
parameter, the code will use default optimal values.
To train a GAN-based model, simply pass --model=DeepGAN
as the command line argument to the commands above.
- Training should run for at least 100000 steps to produce good results. The number of training steps can be set via
--epoch
. - The default learning rate of
1e-4
may not yield great results. Feel free to play with this value to get a well-functioning generator. - The training logic for DeepGAN follows that of caogang's repository. This was done to ease the understanding of my code.
Once training concludes, the trained model will be saved under DeepNAG/logs/unique-session-name/checkpoints
.
The trained model can be visualized using the argument --evaluate
passed to main.py
. Some pretrained models are
included under DeepNAG/pretrained_models
. Run either of the following commands to visualize a trained model's output
(needs Matplotlib):
#
# DeepNAG models
#
# Visualize the pretrained DeepNAG model trained on $1-GDS
python main.py --model=DeepNAG --dataset=dollar-gds --evaluate=pretrained_models/DeepNAG/dollar-gds/checkpoint-best.tar
# Visualize the pretrained DeepNAG model trained on JK2017 (Kinect)
python main.py --model=DeepNAG --dataset=jk2017-kinect --evaluate=pretrained_models/DeepNAG/jk2017-kinect/checkpoint-best.tar
#
# DeepGAN models
#
# Visualize the pretrained DeepGAN model trained on $1-GDS
python main.py --model=DeepGAN --dataset=dollar-gds --evaluate=pretrained_models/DeepGAN/dollar-gds/checkpoint-best.tar
# Visualize the pretrained DeepGAN model trained on JK2017 (Kinect)
python main.py --model=DeepGAN --dataset=jk2017-kinect --evaluate=pretrained_models/DeepGAN/jk2017-kinect/checkpoint-best.tar
Our requirements for this work yielded several other standalone projects which we have also made public.
Our soft DTW for PyTorch in CUDA project, is a fast implementation of the sDTW algorithm on which DeepNAG's loss function relies. We additionally implemented second-order differentiable GRU units for PyTorch using TorchScript (JitGRUs).
If you find our work useful, please consider starring this repository and citing our work:
@phdthesis{maghoumi2020dissertation,
title={{Deep Recurrent Networks for Gesture Recognition and Synthesis}},
author={Mehran Maghoumi},
year={2020},
school={University of Central Florida Orlando, Florida}
}
@misc{maghoumi2020deepnag,
title={{DeepNAG: Deep Non-Adversarial Gesture Generation}},
author={Mehran Maghoumi and Eugene M. Taranta II and Joseph J. LaViola Jr},
year={2020},
eprint={2011.09149},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
This project is licensed under the MIT License - see the LICENSE file for details. Note that JK2017 dataset IS NOT a part of this project, and has a different license.