isabella232 / realtime_handpose_3d

3D_HandPose

This repository implements a realtime 3D hand posture estimation pipeline running on Jetson platform using a Azure Kinect camera.
Please refer to the following repositories before getting started here:

• centernet_kinet
• Hand Posture Estimation

There are 2 stages to our pipeline

• CenterNet Bounding Box

• A2J Posture Detection

• Run inference

CenterNet Bounding Box

The first stage will localize the hand using a fusion of infrared and depth image.
NOTE: more detail can be found in the centernet_kinect repository

A2J Posture Detection

The second stage would perform 3D hand posture estimation on the region of intrest selected by the previous step.
NOTE: for training a model please refer to the Hand Posture Estimation repository

Run inference

• Initially configure the pipeline/constants.py file:
  • CENTERNET_MODEL_PATH please place the centernet model weights in "/checkpoint/CenterNet"
    with the naming convention that was provided in the original repository
    • Configure the centernet portion of the file as its been described in the original repository.
      if you are using the weights directly from the original repository you dont have to modify this section.
  • A2J_MODEL_PATH please place the A2J model weights in "/checkpoint/A2J"
    with the naming convention that was provided in the original repository
    • Configure the a2j portion of the file as you have set up the training pipeline for Hand Posture Estimation.
      
    • For Faster inference we use TensorRT inference engine to optimize the models. this will take some time to compile the models and create a TRT engine
      
• Run realtime inference on a jetson platform.

  cd pipeline
  python3 azure_kinect.py
  
  # Optional for faster inference
  python3 azure_kinect.py --trt True # for optimizing the models with TensorRT fp16