If you find the PoseRBPF code useful, please consider citing:
@inproceedings{deng2019pose,
author = {Xinke Deng and Arsalan Mousavian and Yu Xiang and Fei Xia and Timothy Bretl and Dieter Fox},
title = {PoseRBPF: A Rao-Blackwellized Particle Filter for 6D Object Pose Tracking},
booktitle = {Robotics: Science and Systems (RSS)},
year = {2019}
}
@inproceedings{deng2020self,
author = {Xinke Deng and Yu Xiang and Arsalan Mousavian and Clemens Eppner and Timothy Bretl and Dieter Fox},
title = {Self-supervised 6D Object Pose Estimation for Robot Manipulation},
booktitle = {International Conference on Robotics and Automation (ICRA)},
year = {2020}
}git clone https://github.com/NVlabs/PoseRBPF.git --recursiveInstall dependencies:
- install anaconda according to the official website.
- create the virtual env with
pose_rbpf_env.yml:
conda env create -f pose_rbpf_env.yml
conda activate pose_rbpf_env
- compile the YCB Renderer according to the instruction.
- compile the utility functions with:
sh build.sh
- CAD Model for YCB Objects: 743MB
- CAD Model for T-LESS Objects: 502MB
- PoseCNN weights for 20 YCB Objects: 76MB
- RGB Auto-encoder weights (YCB): 6GB
- RGB-D Auto-encoder weights (YCB): 9GB
- Self-supervided trained RGB Auto-encoder weights (YCB): 5GB
- RGB Auto-encoder weights (T-LESS): 8GB
- RGB-D Auto-encoder weights (T-LESS): 12GB
Downolad files as needed. Extract CAD models under the cad_models directory, and extract model weights under the checkpoints directory.
- The demo shows tracking
003_cracker_boxon YCB Video Dataset. - Run script
download_demo.shto download checkpoint (434 MB), CAD models (743 MB), 2D detections (13 MB), and necessary data (3 GB) for the demo:
./scripts/download_demo.sh
- Then you should have files organized like:
├── ...
├── PoseRBPF
| |── cad_models
| | |── ycb_models
| | └── ...
| |── checkpoints
| | |── ycb_ckpts_roi_rgbd
| | |── ycb_codebooks_roi_rgbd
| | |── ycb_configs_roi_rgbd
| | └── ...
| |── detections
| | |── posecnn_detections
| | |── tless_retina_detections
| |── config # configuration files for training and DPF
| |── networks # auto-encoder networks
| |── pose_rbpf # particle filters
| └── ...
|── YCB_Video_Dataset # to store ycb data
| |── cameras
| |── data
| |── image_sets
| |── keyframes
| |── poses
| └── ...
└── ...
- Run demo with
003_cracker_box. The results will be stored in./results/
./scripts/run_demo.sh
- Due to the incompatibility between ROS Kinetic and Python 3, the ROS node only runs with Python 2.7. We first create the virtual env with
pose_rbpf_env_py2.yml:
conda env create -f pose_rbpf_env_py2.yml
conda activate pose_rbpf_env_py2
- compile the YCB Renderer according to the instruction.
- compile the utility functions with:
sh build.sh
- Make sure you can run the demo above first.
- Install ROS if it's not there:
sudo sh -c 'echo "deb http://packages.ros.org/ros/ubuntu $(lsb_release -sc) main" > /etc/apt/sources.list.d/ros-latest.list'
sudo apt-key adv --keyserver 'hkp://keyserver.ubuntu.com:80' --recv-key C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654
sudo apt-get update
sudo apt-get install ros-kinetic-desktop-full
- Update python packages:
conda install -c auto catkin_pkg
pip install -U rosdep rosinstall_generator wstool rosinstall six vcstools
pip install msgpack
pip install empy
- Source ROS (every time before launching the node):
source /opt/ros/kinetic/setup.bash
- Initialze rosdep:
sudo rosdep init
rosdep update
- Download demo rosbag:
./scripts/download_ros_demo.sh
- Run PoseCNN node (with roscore running in another terminal, download PoseCNN weights first):
./scripts/run_ros_demo_posecnn.sh
- Run PoseRBPF node for RGB-D tracking (with roscore running in another terminal):
./scripts/run_ros_demo.sh
- (Optional) For RGB tracking run this command instead:
./scripts/run_ros_demo_rgb.sh
- Run RVIZ in the PoseRBPF directory:
rosrun rviz rviz -d ./ros/tracking.rviz
- Once you see
*** PoseRBPF Ready ...in the PoseRBPF terminal, run rosbag in another terminal, then you should be able to see the tracking demo:
rosbag play ./ros_data/demo_single.bag
- Download demo rosbag:
./scripts/download_ros_demo_multiple.sh
- Run PoseCNN node (with roscore running in another terminal, download PoseCNN weights first):
./scripts/run_ros_demo_posecnn.sh
- Run PoseRBPF node with self-supervised trained RGB Auto-encoder weights:
./scripts/run_ros_demo_rgb_multiple_ssv.sh
- (Optional) Run PoseRBPF node with RGB-D Auto-encoder weights instead:
./scripts/run_ros_demo_multiple.sh
- (Optional) Run PoseRBPF node with RGB Auto-encoder weights instead:
./scripts/run_ros_demo_rgb_multiple.sh
- Run RVIZ in the PoseRBPF directory:
rosrun rviz rviz -d ./ros/tracking.rviz
- Once you see
*** PoseRBPF Ready ...in the PoseRBPF terminal, run rosbag in another terminal, then you should be able to see the tracking demo:
rosbag play ./ros_data/demo_multiple.bag
Note that PoseRBPF takes certain time to initialize each object before tracking. You can pause the ROS bag by pressing space for initialization, and then press space again to resume tracking.
- Download checkpoints from the google drive folder (
ycb_rgbd_full.tar.gzorycb_rgb_full.tar.gz) and unzip to the checkpoint directory. - Download all the data in the YCB Video Dataset so the
../YCB_Video_Dataset/datafolder contains all the sequences. - Run RGB-D tracking (use
002_master_chef_canas an example here):
sh scripts/test_ycb_rgbd/val_ycb_002_rgbd.sh 0 1
- Run RGB tracking (use
002_master_chef_canas an example here):
sh scripts/test_ycb_rgb/val_ycb_002_rgb.sh 0 1
- Download checkpoints from the google drive folder (
tless_rgbd_full.tar.gzortless_rgb_full.tar.gz) and unzip to the checkpoint directory. - Download all the data in the T-LESS Dataset so the
../TLess/folder contains all the sequences. - Download all the models for T-LESS objects from the google drive folder.
- Then you should have files organized like:
├── ...
├── PoseRBPF
| |── cad_models
| | |── ycb_models
| | |── tless_models
| | └── ...
| |── checkpoints
| | |── tless_ckpts_roi_rgbd
| | |── tless_codebooks_roi_rgbd
| | |── tless_configs_roi_rgbd
| | └── ...
| |── detections
| | |── posecnn_detections
| | |── tless_retina_detections
| |── config # configuration files for training and DPF
| |── networks # auto-encoder networks
| |── pose_rbpf # particle filters
| └── ...
|── YCB_Video_Dataset # to store ycb data
| |── cameras
| |── data
| |── image_sets
| |── keyframes
| |── poses
| └── ...
|── TLess # to store tless data
| |── t-less_v2
|── tless_ckpts_roi_rgbd
| | |── test_primesense
| | └── ...
| └── ...
└── ...
- Run RGB-D tracking (use
obj_01as an example here):
sh scripts/test_tless_rgbd/val_tless_01_rgbd.sh 0 1
- Run RGB tracking (use
obj_01as an example here):
sh scripts/test_tless_rgb/val_tless_01_rgb.sh 0 1
-
Download checkpoints from the google drive folder (
ycb_rgbd_full.tar.gzorycb_rgb_full.tar.gz) and unzip to the checkpoint directory. -
Download the DexYCB dataset from here.
-
Download PoseCNN results on the DexYCB dataset from here.
-
Create a symlink for the DexYCB dataset and the PoseCNN results
cd $ROOT/data/DEX_YCB ln -s $dex_ycb_data data ln -s $results_posecnn_data results_posecnn
-
Install PyTorch PoseCNN layers according to the instructions here.
-
Run RGB-D tracking:
./scripts/test_dex_rgbd/dex_ycb_test_rgbd_s0.sh $GPU_ID -
Run RGB tracking:
./scripts/test_dex_rgb/dex_ycb_test_rgb_s0.sh $GPU_ID
- Download microsoft coco dataset 2017 val images from here for data augmentation.
- Store the folder
val2017in../coco/ - Run training example for
002_master_chef_canin the YCB objects. The training should be able to run on one single NVIDIA TITAN Xp GPU:
sh scripts/train_ycb_rgbd/train_script_ycb_002.sh
We have referred to part of the RoI align code from maskrcnn-benchmark.
PoseRBPF is licensed under the NVIDIA Source Code License - Non-commercial.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent