kami-code / dexart-release

DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects, CVPR 2023

DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects

[Project Page] [arXiv] [Paper]

DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects,

Chen Bao*, Helin Xu*, Yuzhe Qin, Xiaolong Wang, CVPR 2023.

DexArt is a novel benchmark and pipeline for learning multiple dexterous manipulation tasks. This repo contains the simulated environment and training code for DexArt.

News

[2023.11.21] All the RL checkpoints are available now!🎈 They are included in the assets. See Main Results to reproduce the results in the paper!
[2023.4.18] Code and vision pre-trained models are available now!
[2023.3.24] DexArt is accepted by CVPR 2023! 🎉

Installation

1. Clone the repo and Create a conda env with all the Python dependencies.

git clone [email protected]:Kami-code/dexart-release.git
cd dexart-release
conda create --name dexart python=3.8
conda activate dexart
pip install -e .    # for simulation environment
conda install pytorch==1.12.1 torchvision==0.13.1 torchaudio==0.12.1 -c pytorch    # for visualizing trained policy and training 

2. Download the assets from the Google Drive and place the asset directory at the project root directory.

File Structure

The file structure is listed as follows:

dexart/env/: environments

assets/: tasks annotations, object, robot URDFs and RL checkpoints

examples/: example code to try DexArt

stable_baselines3/: RL training code modified from stable_baselines3

Quick Start

Example of Random Action

python examples/random_action.py --task_name=laptop

task_name: name of the environment [faucet, laptop, bucket, toilet]

Example for Visualizing Point Cloud Observation

python examples/visualize_observation.py --task_name=laptop

task_name: name of the environment [faucet, laptop, bucket, toilet]

Example for Visualizing Policy

python examples/visualize_policy.py --task_name=laptop --checkpoint_path assets/rl_checkpoints/laptop/laptop_nopretrain_0.zip

task_name: name of the environment [faucet, laptop, bucket, toilet]

use_test_set: flag to determine evaluating with seen or unseen instances

Example for Training RL Agent

python3 examples/train.py --n 100 --workers 10 --iter 5000 --lr 0.0001 &&
--seed 100 --bs 500 --task_name laptop --extractor_name smallpn &&
--pretrain_path ./assets/vision_pretrain/laptop_smallpn_fulldata.pth 

n: the number of rollouts to be collected in a single episode

workers: the number of simulation progress

iter: the total episode number to be trained

lr: learning rate of RL

seed: seed of RL

bs: batch size of RL update

task_name: name of training environment [faucet, laptop, bucket, toilet]

extractor_name: different PointNet architectures [smallpn, meduimpn, largepn]

pretrain_path: path to downloaded pre-trained model. [Default: None]

save_freq: save the model every save_freq episodes. [Default: 1]

save_path: path to save the model. [Default: ./examples]

Main Results

python examples/evaluate_policy.py --task_name=laptop --checkpoint_path assets/rl_checkpoints/laptop/laptop_nopretrain_0.zip --eval_per_instance 100
python examples/evaluate_policy.py --task_name=laptop --use_test_set --checkpoint_path assets/rl_checkpoints/laptop/laptop_nopretrain_0.zip --eval_per_instance 100

task_name: name of the environment [faucet, laptop, bucket, toilet]

use_test_set: flag to determine evaluating with seen or unseen instances

Faucet

Method	Split	Seed 0	Seed 1	Seed 2	Avg	Std
No Pre-train	train/test	0.52/0.34	0.00/0.00	0.44/0.43	0.32/0.26	0.23/0.18
Segmentation on PMM	train/test	0.42/0.38	0.25/0.15	0.14/0.11	0.27/0.21	0.11/0.12
Classification on PMM	train/test	0.40/0.33	0.19/0.14	0.07/0.09	0.22/0.18	0.14/0.10
Reconstruction on DAM	train/test	0.27/0.17	0.37/0.30	0.36/0.21	0.33/0.22	0.05/0.05
SimSiam on DAM	train/test	0.80/0.60	0.40/0.24	0.72/0.53	0.64/0.46	0.17/0.16
Segmentation on DAM	train/test	0.80/0.56	0.76/0.53	0.82/0.66	0.79/0.59	0.02/0.05

Laptop

Method	Split	Seed 0	Seed 1	Seed 2	Avg	Std
No Pre-train	train/test	0.78/0.41	0.78/0.31	0.81/0.50	0.79/0.41	0.02/0.08
Segmentation on PMM	train/test	0.91/0.62	0.90/0.53	0.77/0.48	0.86/0.54	0.06/0.08
Classification on PMM	train/test	0.96/0.51	0.58/0.35	0.96/0.62	0.83/0.49	0.18/0.11
Reconstruction on DAM	train/test	0.85/0.56	0.91/0.63	0.80/0.43	0.85/0.54	0.05/0.08
SimSiam on DAM	train/test	0.84/0.59	0.83/0.34	0.89/0.51	0.86/0.48	0.03/0.10
Segmentation on DAM	train/test	0.89/0.57	0.94/0.67	0.89/0.58	0.91/0.60	0.02/0.04

Bucket

Method	Split	Seed 0	Seed 1	Seed 2	Avg	Std
No Pre-train	train/test	0.36/0.55	0.58/0.69	0.52/0.49	0.49/0.57	0.09/0.08
Segmentation on PMM	train/test	0.62/0.62	0.00/0.00	0.40/0.41	0.34/0.34	0.26/0.26
Classification on PMM	train/test	0.55/0.47	0.50/0.51	0.67/0.73	0.57/0.57	0.07/0.11
Reconstruction on DAM	train/test	0.49/0.49	0.58/0.46	0.40/0.59	0.49/0.51	0.07/0.05
SimSiam on DAM	train/test	0.00/0.00	0.53/0.38	0.73/0.78	0.42/0.39	0.30/0.32
Segmentation on DAM	train/test	0.70/0.68	0.70/0.74	0.79/0.85	0.73/0.75	0.04/0.07

Toilet

Method	Split	Seed 0	Seed 1	Seed 2	Avg	Std
No Pre-train	train/test	0.80/0.47	0.75/0.51	0.63/0.43	0.72/0.47	0.07/0.03
Segmentation on PMM	train/test	0.78/0.42	0.62/0.46	0.64/0.47	0.68/0.45	0.07/0.02
Classification on PMM	train/test	0.78/0.33	0.65/0.43	0.66/0.44	0.69/0.40	0.06/0.05
Reconstruction on DAM	train/test	0.78/0.58	0.73/0.48	0.75/0.49	0.75/0.52	0.02/0.05
SimSiam on DAM	train/test	0.84/0.54	0.81/0.49	0.84/0.45	0.83/0.50	0.01/0.04
Segmentation on DAM	train/test	0.86/0.54	0.84/0.53	0.86/0.56	0.85/0.54	0.01/0.01

Visual Pretraining

We have uploaded the code to generate a dataset and pretrain our models in examples/pretrain. You can refer to examples/pretrain/run.sh for a detailed usage.

Bibtex

@inproceedings{bao2023dexart,
  title={DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects},
  author={Bao, Chen and Xu, Helin and Qin, Yuzhe and Wang, Xiaolong},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={21190--21200},
  year={2023}
}

Acknowledgements

This repository employs the same code structure for simulation environment and training code to that used in DexPoint.

dexart-release's Issues

How to produce the task annotation like the project provided?

Hi, @Kami-code

Thanks for the great work :)
How can I produce the task annotation like the project provided (e.g., xxx_joint_annotation.json, xxx_scale.json, xxx_allegro_hand_xarm6_wrist_mounted_face_front_relative_position.json) for other objects from sapien?

Thanks for any help:)

Consultation about Figure 9 in the paper

Hi, @Kami-code

Thanks for this great work!

When I look backward at Figure 9, some details are confusing me:(

1. Do you use all instances of laptops or only one instance? It seems to plot about 35 sampling camera positions.
2. Should I fix the pose of the instance to calculate the radius r of the semi-sphere?
3. Is it convenient to open source the code for sampling 35 camera poses, I would like to verify the robustness of the algorithm by using the benchmark.

Thanks for any help:)
Best,
Charlie

"SAPIEN is not responding"

And I have another question,when we run python examples/visualize_observation.py --task_name=laptop,is it dynamic or static?
mine is static,and a few seconds later,"SAPIEN is not responding"occurs.

How to draw the visual point cloud simulation like in Project?

Hi, @Kami-code

Thanks for the wonderful work.

Could you please open-source the code to showcase the video for the visual point cloud simulation like in Project?

Thanks for any help:)

assets文件夹没有呢，在哪里获取？

Multiprocessing issue? Advice on running multiple trains at the same time

Hi, I have been trying to run a few training at the same time using the example code given in the README.md. I noticed that some of the runs take a very long time, almost looking like it is stuck. Therefore, I went to check the fps in the tensorboard log.

I started one training at 17:37 and then another at 17:42. I noticed that there is a drop in fps (blue line) at 17:42, exactly when the second train is started. It seems like there is an issue with multiprocessing. When I run more trains at the same time, the drop in fps is even sharper.

I have checked my cpu and gpu resources, they aren't full. top command also shows that there are unused cpu capacity, even when some of the trains are stuck.

May I know is it possible to run multiple trains at the same time? If yes, can you please advise on how to estimate the maximum number of trains I can run?

Here are some details of the server I used:
GPU: NVIDIA GeForce RTX 3090 * 4
CPU: 60 cores
Memory: 320 GB

missing assets? failed to load visual mesh & random lengthy training

Hi.
It seems like there is an issue with the textured objects of 9918 and 10090, it reports "Failed to load visual mesh" when running random_action.py. Can you please check on this issue?

My temporary solution is to modify the task setting and omit these 2 objects, does this potentially lead to training issues? I have been experiencing random lengthy training epochs that takes a long time on env.step(), which almost looks like it's stuck in a loop.

Thank you in advance for the help!

Reproducibility

Hi I tested your evaluation script but it seems that it is non-deterministic even if testing with the same exact checkpoint
Running the following command 3 times:
python examples/evaluate_policy.py --task_name=laptop --checkpoint_path assets/rl_checkpoints/laptop.zip --eval_per_instance 10
I obtain the following results

1. reward mean: 227 success rate: 0.57
2. reward mean: -0.12 success rate: 0.48
3. reward mean: -0.053 success rate: 0.38
   Is it normal that the evaluation changes across different evaluations? the policy is always the same.
   Also the results for the laptop task don't match the results of the paper, am I doing something wrong?

assets questions

The assest dataset is missing some files, resulting in code running errors, is there a complete assets dataset.

Missing obj in assets

Hello, congratulations on your excellent work. I am trying to run the code, but when I followed the steps in readme.md, I encountered the following bug. It seems that the assets compressed package does not contain all the object files, can you please update the file? In addition, is it convenient to give me some details about the hardware and experiment configuration of the training policy? I did not find a description in the paper. Thank you. Best wishes.

Some issue about model saving

When I use save function of policy: self.policy.save()，I encounter a issue as following:
"_generator_ctor() takes from 0 to 1 positional arguments but 2 were given."
However, when i use the saving function of policy without 'data':self_get_constructor_parameters(), It works fine.
Can I just ignore the 'data':self_get_constructor_parameters() when saving the model?

Visual pretraining code and dataset

Hi,

Thank you for your work. The paper mentions that you have a DAM dataset of 6k point clouds which was used to pretrain your pointNet. Is that dataset available anywhere and can you point me to the code for pretraining the encoder?

Thanks!

Training details

Hi thank you for your work, I would like to reproduce the experiments reported in the paper, could you provide your training details for the experiments reported in table 2?

Also, could you provide a link for the supplementary material, I'm having trouble finding it online but it is referenced in the paper.

NameError: name 'freeze_type' is not defined

Thanks for your great work,I have also met some problems ,it seems that new-1.obj had missed,and when I tried
Traceback (most recent call last):
File "examples/train.py", line 111, in
if freeze_type > 0:
NameError: name 'freeze_type' is not defined
it affect the training.

How to evaluate the Success Rate like Table 2 in the paper

Hi, @Kami-code

Thanks for the wonderful work.
How can I evaluate the success rate like Table 2 in the paper?

Thanks for any help:)

Reason for implementing a different env class for each task despite the high similarity

Hi, thanks for your amazing work!
I have been checking out this project. Based on my understanding, it seems like the RL environment for the tasks are quite similar (esp. for laptop and toilet), may I know what's your consideration when you choose to implement a new class for each of these task environment instead of using just the base class and modify the config?

Assets Google Drive Download Link Missing

Hi! Great work of yours.

I'm trying to follow the instructions to reproduce your work, but the download link in Google Drive seems to be missing. Could it be that the link has expired?