zhrmichael-logm / neural-retargeting

Code for the paper "Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language", RAL with ICRA 2022

Neural Retargeting

Code for the paper "Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language"

Prerequisite

• PyTorch Tensors and Dynamic neural networks in Python with strong GPU acceleration
• pytorch_geometric Geometric Deep Learning Extension Library for PyTorch
• Kornia a differentiable computer vision library for PyTorch.
• HDF5 for Python The h5py package is a Pythonic interface to the HDF5 binary data format.

Dataset

The Chinese sign language dataset can be downloaded here.

Model

The pretrained model can be downloaded here.

Get Started

Training

CUDA_VISIBLE_DEVICES=0 python main.py --cfg './configs/train/yumi.yaml'

Inference

CUDA_VISIBLE_DEVICES=0 python inference.py --cfg './configs/inference/yumi.yaml'

Simulation Experiment

We build the simulation environment using pybullet, and the code is in this repository.

After inference is done, the motion retargeting results are stored in a h5 file. Then run the sample code here.

Real-World Experiment

Real-world experiments could be conducted on ABB's YuMi dual-arm collaborative robot equipped with Inspire-Robotics' dexterous hands.

We release the code in this repository, please follow the instructions.

Citation

If you find this project useful in your research, please cite this paper.

@article{zhang2022kinematic,
  title={Kinematic Motion Retargeting via Neural Latent Optimization for Learning Sign Language},
  author={Zhang, Haodong and Li, Weijie and Liu, Jiangpin and Chen, Zexi and Cui, Yuxiang and Wang, Yue and Xiong, Rong},
  journal={IEEE Robotics and Automation Letters},
  year={2022},
  publisher={IEEE}
}