nvlabs / handover-sim

A simulation environment and benchmark for human-to-robot object handovers

Handover-Sim

Handover-Sim is a Python-based simulation environment and benchmark for human-to-robot object handovers. The environment and benchmark were initially described in an ICRA 2022 paper:

HandoverSim: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers
Yu-Wei Chao, Chris Paxton, Yu Xiang, Wei Yang, Balakumar Sundaralingam, Tao Chen, Adithyavairavan Murali, Maya Cakmak, Dieter Fox
IEEE International Conference on Robotics and Automation (ICRA), 2022
[ paper ] [ video ] [ arXiv ] [ project site ]

Citing Handover-Sim

@INPROCEEDINGS{chao:icra2022,
  author    = {Yu-Wei Chao and Chris Paxton and Yu Xiang and Wei Yang and Balakumar Sundaralingam and Tao Chen and Adithyavairavan Murali and Maya Cakmak and Dieter Fox},
  booktitle = {IEEE International Conference on Robotics and Automation (ICRA)},
  title     = {{HandoverSim}: A Simulation Framework and Benchmark for Human-to-Robot Object Handovers},
  year      = {2022},
}

License

Handover-Sim is released under the BSD 3-Clause License.

Acknowledgements

This repo is based on a Python project template created by Rowland O'Flaherty.

Contents

1. Prerequisites
2. Installation
3. Running Demos
4. Benchmarking Baselines
   1. Yang et al. ICRA 2021
   2. OMG Planner
   3. GA-DDPG
5. Evaluation
6. Reproducing ICRA 2022 Results
7. Rendering from Result and Saving Rendering

Prerequisites

This code is tested with Python 3.8 on Ubuntu 20.04.

Installation

For good practice for Python package management, it is recommended to install the package into a virtual environment (e.g., virtualenv or conda).

1. Clone the repo with --recursive and and cd into it:

   git clone --recursive https://github.com/NVlabs/handover-sim.git
   cd handover-sim

2. Install handover-sim and submodule mano_pybullet as Python packages:

   pip install -e .
   pip install --no-deps -e ./mano_pybullet

3. Download MANO models and code (mano_v1_2.zip) from the MANO website and place the file under handover/data/. Unzip with:

   cd handover/data
   unzip mano_v1_2.zip
   cd ../..

   This will extract a folder handover/data/mano_v1_2/.

4. Download the DexYCB dataset.

   Option 1: Download cached dataset: (recommended)

   1. Download dex-ycb-cache-20220323.tar.gz (507M) and place the file under handover/data/. Extract with:

      cd handover/data
      tar zxvf dex-ycb-cache-20220323.tar.gz
      cd ../..

      This will extract a folder handover/data/dex-ycb-cache/.

   Option 2: Download full dataset and cache the data:

   1. Download the DexYCB dataset from the DexYCB project site.

   2. Set the environment variable for dataset path:

      export DEX_YCB_DIR=/path/to/dex-ycb

      $DEX_YCB_DIR should be a folder with the following structure:

      ├── 20200709-subject-01/
      ├── 20200813-subject-02/
      ├── ...
      ├── calibration/
      └── models/

   3. Cache the dataset:

      python handover/data/cache_dex_ycb_data.py

      The cached dataset will be saved to handover/data/dex-ycb-cache/.

5. Compile assets.

   1. Download assets-3rd-party-20220511.tar.gz (155M) and place the file under handover/data/. Extract with:

      cd handover/data
      tar zxvf assets-3rd-party-20220511.tar.gz
      cd ../..

      This will extract a folder handover/data/assets/ with 3rd party assets. See handover/data/README.md for the source of these assets.

   2. Compile assets:

      ./handover/data/compile_assets.sh

      The compiled assets will be saved to handover/data/assets/.

Running Demos

1. Running a handover environment:

   python examples/demo_handover_env.py \
     SIM.RENDER True

   

2. Running a planned trajectory:

   python examples/demo_trajectory.py \
     SIM.RENDER True

   

3. Running a benchmark wrapper:

   python examples/demo_benchmark_wrapper.py \
     SIM.RENDER True \
     BENCHMARK.DRAW_GOAL True

   

   This will run the same trajectory as in demo_trajectory.py above but will also draw the goal region in the visualizer window and print out the benchmark status in the terminal.

Benchmarking Baselines

We benchmarked three baselines on Handover-Sim:

1. OMG Planner - GitHub
2. Yang et al. ICRA 2021 - arXiv
3. GA-DDPG - GitHub

OMG Planner	Yang et al. ICRA 2021

GA-DDPG (hold)	GA-DDPG (w/o hold)

As described in the paper Sec. IV "Training and Evaluation Setup", we divide the data into different setups (s0, s1, s2, s3) and splits (train, val, test). We benchmarked these baselines on the test split of each setup.

Below we provide instructions for setting up and running benchmark for these baselines.

Yang et al. ICRA 2021

• We have included our implementation of Yang et al. ICRA 2021 in this repo. The following command will run the benchmark on the test split of s0:

  python examples/run_benchmark_yang_icra2021.py \
    SIM.RENDER True \
    BENCHMARK.SETUP s0

  This will open a visualizer window, go through each handover scene in the split, and execute the actions generated from the policy. To run on other setups, replace s0 with s1, s2, and s3.

• The command above is mostly just for visualization purposes, and thus does not save the benchmark result. To save the result for evaluation later, set BENCHMARK.SAVE_RESULT to True, and remove SIM.RENDER to run headless if you don't need the visualizer window:

  python examples/run_benchmark_yang_icra2021.py \
    BENCHMARK.SETUP s0 \
    BENCHMARK.SAVE_RESULT True

  The result will be saved to a new folder results/*_yang-icra2021_*_test/.

• Once the job finishes, you are ready to run evaluation and see the result. See the Evaluation section.

OMG Planner

• First, you need to install OMG-Planner. See examples/README.md for our documentation for installation steps.

• Once installed, you can run the benchmark on the test split of s0 with the path to OMG-Planner (OMG_PLANNER_DIR):

  OMG_PLANNER_DIR=OMG-Planner python examples/run_benchmark_omg_planner.py \
    SIM.RENDER True \
    BENCHMARK.SETUP s0

  Like in Yang et al. ICRA 2021, this will open a visualizer window, go through each handover scene in the split, and execute the actions generated from the policy. To run on other setups, replace s0 with s1, s2, and s3.

• Likewise, the command above is mostly just for visualization purposes, and thus does not save the benchmark result. To save the result for evaluation later, set BENCHMARK.SAVE_RESULT to True, and remove SIM.RENDER to run headless if you don't need the visualizer window:

  OMG_PLANNER_DIR=OMG-Planner python examples/run_benchmark_omg_planner.py \
    BENCHMARK.SETUP s0 \
    BENCHMARK.SAVE_RESULT True

  The result will be saved to a new folder results/*_omg-planner_*_test/.

• Again, once the job finishes, you are ready to run evaluation and see the result. See the Evaluation section.

GA-DDPG

• First, you need to install GA-DDPG. See examples/README.md for our documentation for installation steps.

• Once installed, you can run the benchmark. As described in the paper Sec. V "Baselines", we benchmarked two variants of this baseline:

  • Hold until the human hand stops as in the OMG Planner ("GA-DDPG hold")
  • Without hold ("GA-DDPG w/o hold")

• With the path to GA-DDPG (GADDPG_DIR), you can now run for "GA-DDPG hold" on the test split of s0 with:

  GADDPG_DIR=GA-DDPG CUDA_VISIBLE_DEVICES=0 python examples/run_benchmark_gaddpg_hold.py \
    SIM.RENDER True \
    ENV.ID HandoverHandCameraPointStateEnv-v1 \
    BENCHMARK.SETUP s0

  and for "GA-DDPG w/o hold" on the test split of s0 with:

  GADDPG_DIR=GA-DDPG CUDA_VISIBLE_DEVICES=0 python examples/run_benchmark_gaddpg_wo_hold.py \
    SIM.RENDER True \
    ENV.ID HandoverHandCameraPointStateEnv-v1 \
    BENCHMARK.SETUP s0

  Like in Yang et al. ICRA 2021, this will open a visualizer window, go through each handover scene in the split, and execute the actions generated from the policy. To run on other setups, replace s0 with s1, s2, and s3.

• Note that different than in Yang et al. ICRA 2021 and OMG Planner, we explicitly set ENV.ID to HandoverHandCameraPointStateEnv-v1 in the commands above.

  • HandoverHandCameraPointStateEnv-v1 specifies a different environemnt than the default HandoverStateEnv-v1 used in Yang et al. ICRA 2021 and OMG Planner.
  • HandoverHandCameraPointStateEnv-v1 provides the point cloud input used in GA-DDPG, while HandoverStateEnv-v1 provides ground-truth state information of which Yang et al. ICRA 2021 and OMG Planner can directly consume.

• Likewise, the command above is mostly just for visualization purposes, and thus does not save the benchmark result. To save the result for evaluation later, set BENCHMARK.SAVE_RESULT to True, and remove SIM.RENDER to run headless if you don't need the visualizer window. For "GA-DDPG hold", run:

  GADDPG_DIR=GA-DDPG CUDA_VISIBLE_DEVICES=0 python examples/run_benchmark_gaddpg_hold.py \
    ENV.ID HandoverHandCameraPointStateEnv-v1 \
    BENCHMARK.SETUP s0 \
    BENCHMARK.SAVE_RESULT True

  and for "GA-DDPG w/o hold", run:

  GADDPG_DIR=GA-DDPG CUDA_VISIBLE_DEVICES=0 python examples/run_benchmark_gaddpg_wo_hold.py \
    ENV.ID HandoverHandCameraPointStateEnv-v1 \
    BENCHMARK.SETUP s0 \
    BENCHMARK.SAVE_RESULT True

  The result will be saved to a new folder results/*_ga-ddpg-hold_*_test/ for "GA-DDPG hold" and results/*_ga-ddpg-wo-hold_*_test/ for "GA-DDPG w/o hold".

• Again, once the job finishes, you are ready to run evaluation and see the result. See the Evaluation section.

Evaluation

• To evaluate the result of a baseline, all you need is the result folder generated from running the benchmark. For example, if your result folder is results/2022-02-28_08-57-34_yang-icra2021_s0_test/, run the following command:

  python examples/evaluate_benchmark.py \
    --res_dir results/2022-02-28_08-57-34_yang-icra2021_s0_test

  You should see an output similar to the following in the terminal:

  2022-06-03 16:13:46: Running evaluation for results/2022-02-28_08-57-34_yang-icra2021_s0_test
  2022-06-03 16:13:47: Evaluation results:
  |  success rate   |    mean accum time (s)    |                    failure (%)                     |
  |      (%)        |  exec  |  plan  |  total  |  hand contact   |   object drop   |    timeout     |
  |:---------------:|:------:|:------:|:-------:|:---------------:|:---------------:|:--------------:|
  | 64.58 ( 93/144) | 4.864  | 0.036  |  4.900  | 17.36 ( 25/144) | 11.81 ( 17/144) | 6.25 (  9/144) |
  2022-06-03 16:13:47: Printing scene ids
  2022-06-03 16:13:47: Success (93 scenes):
  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---
    0    1    2    3    4    5    6    7    8    9   10   12   13   15   16   17   18   19   21   22
   23   25   26   27   28   30   33   34   35   36   37   38   42   43   46   49   50   53   54   56
   59   60   62   63   64   66   68   69   70   71   72   77   81   83   85   87   89   91   92   93
   94   95   96   98  103  106  107  108  109  110  111  112  113  114  115  116  117  120  121  123
  125  126  127  128  130  131  132  133  137  138  139  141  143
  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---
  2022-06-03 16:13:47: Failure - hand contact (25 scenes):
  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---
   11   14   20   29   39   40   41   44   45   47   51   55   57   58   65   67   74   80   82   88
  102  105  118  124  136
  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---
  2022-06-03 16:13:47: Failure - object drop (17 scenes):
  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---
   24   31   32   52   61   78   79   84   86   97  101  104  119  122  134  140  142
  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---  ---
  2022-06-03 16:13:47: Failure - timeout (9 scenes):
  ---  ---  ---  ---  ---  ---  ---  ---  ---
   48   73   75   76   90   99  100  129  135
  ---  ---  ---  ---  ---  ---  ---  ---  ---
  2022-06-03 16:13:47: Evaluation complete.
  

  The same output will also be logged to results/2022-02-28_08-57-34_yang-icra2021_s0_test/evaluate.log.

• To benchmark and evaluate your own method, you need to first generate a result folder in the same format.

  • It is advisable to use the BenchmarkRunner class in handover/benchmark_runner.py to benchmark and generate the result folder for your method.
  • You can follow the example of examples/run_benchmark_omg_planner.py.

Reproducing ICRA 2022 Results

We provide the result folders of the benchmarks reported in the ICRA 2022 paper. You can run evaluation on these files and reproduce the exact numbers in the paper.

To run the evaluation, you need to first download the ICRA 2022 results.

./results/fetch_icra2022_results.sh

This will extract a folder results/icra2022_results/ containing the result folders.

You can now run evaluation on these result folders. For example, for Yang et al. ICRA 2021 on s0, run:

python examples/evaluate_benchmark.py \
  --res_dir results/icra2022_results/2022-02-28_08-57-34_yang-icra2021_s0_test

You should see the exact same result shown in the example of the Evaluation section.

The full set of evaluation commands can be found in examples/all_icra2022_results_eval.sh.

Rendering from Result and Saving Rendering

• While you can run a benchmark with a visualizer window by adding SIM.RENDER True (e.g., see Yang et al. ICRA 2021), you can also run headless and re-render the rollouts with a visualizer window after the fact—as long as you saved the result with BENCHMARK.SAVE_RESULT True.

  For example, if your result folder is results/2022-02-28_08-57-34_yang-icra2021_s0_test/, run the following command:

  python examples/render_benchmark.py \
    --res_dir results/2022-02-28_08-57-34_yang-icra2021_s0_test \
    SIM.RENDER True

  This will run the same benchmark environment with a policy that simply loads and executes the actions from the saved result.

  Consequently, if you have downloaded the ICRA 2022 results following the Reproducing ICRA 2022 Results Section, you can also try rendering from one of the downloaded result folders, for example:

  python examples/render_benchmark.py \
    --res_dir results/icra2022_results/2022-02-28_08-57-34_yang-icra2021_s0_test \
    SIM.RENDER True

  This allows you to visualize the rollouts in the ICRA 2022 results.

• Apart from the visualizer window, you can also re-render the rollouts with an offscreen renderer and further save the rendered frame into .jpg files. These .jpg files can later further be converted into .mp4 video files for offline visualization.

  For example, if your result folder is results/2022-02-28_08-57-34_yang-icra2021_s0_test/, run the following command:

  python examples/render_benchmark.py \
    --res_dir results/2022-02-28_08-57-34_yang-icra2021_s0_test \
    ENV.RENDER_OFFSCREEN True \
    BENCHMARK.SAVE_OFFSCREEN_RENDER True

  This will save the offscreen rendered frames to folders named after the scene ID (e.g., 000/, 001/, etc.) under results/2022-02-28_08-57-34_yang-icra2021_s0_test/. Each folder contains the rendered frames of one scene.

  By default, the offscreen rendering will use Bullet's CPU based TinyRenderer, which may take a while to run. If you have a GPU, you may speed up rendering by using Bullet's hardware accelerated OpenGL rendering with EGL. If your result folder is results/2022-02-28_08-57-34_yang-icra2021_s0_test/, you can run:

  ./examples/render_benchmark_egl.sh results/2022-02-28_08-57-34_yang-icra2021_s0_test

  Warning: Rendering frames for a full test split with hundreds of scenes may still take a couple of hours even with the GPU based OpenGL rendering.

  Finally, once you have finished rendering the .jpg files for all the scenes, you can easily convert .jpg to .mp4 with ffmpeg. First, install ffmpeg if you have not, for example, with sudo apt install ffmpeg.

  If your result folder is results/2022-02-28_08-57-34_yang-icra2021_s0_test/, you can then run:

  ./examples/generate_mp4_from_jpg.sh results/2022-02-28_08-57-34_yang-icra2021_s0_test

  This will generate .mp4 files named after the scene ID (e.g., 000.mp4, 001.mp4, etc.) under results/2022-02-28_08-57-34_yang-icra2021_s0_test/. Each .mp4 is converted from the .jpg files of one scene.

  

handover-sim's Issues

How to introduce RGBD image to the policy network for the handover_sim environment

I want to obtain the RGBD as state and input to the policy network, but I find the simulator rendering became very slow. I mainly modify three prats of the code.

Specifically, I modify part of the code in the project, including handover_env.py (class HandoverHandCameraPointStateEnv(HandoverEnv) ) and run_benchmark_gaddpg_hold.py (class GADDPGPolicy(SimplePolicy) and class PointListener).

1. I want to introduce the rgb and depth image to the network, so add two more items to the observation, i.e., observation["rgb"] and observation["depth"], as following:

class HandoverHandCameraPointStateEnv(HandoverEnv) in handover_env.py with modification details:

def _get_observation(self):
    observation = {}
    observation["frame"] = self.frame
    observation["panda_link_ind_hand"] = self.panda.LINK_IND_HAND
    observation["panda_body"] = self.panda.body
    observation["callback_get_point_state"] = self._get_point_state
    **observation["rgb"]=self.panda._camera.color[0]
    observation["depth"]=self.panda._camera.depth[0]**
    return observation

2. Use obs["rgb"] and obs["depth"] to the policy network, with corresponding modification details:

class GADDPGPolicy(SimplePolicy) in run_benchmark_gaddpg_hold.py

def plan(self, obs):
    info = {}

    if (obs["frame"] - self._steps_wait) % self._steps_action_repeat == 0:
        point_state, obs_time = self._get_point_state_from_callback(obs)
        info["obs_time"] = obs_time

        if point_state.shape[1] == 0 and self._point_listener.acc_points.shape[1] == 0:
            action = np.array(self._cfg.ENV.PANDA_INITIAL_POSITION)
        else:
            ef_pose = self._get_ef_pose(obs)
            ef_pose_new = self._point_listener.run_network(**(point_state, obs["rgb"], obs["depth"])**, ef_pose)
       ...
       ...

3. rgb and depth concat together and resize to (112, 112), with corresponding modification details:

class PointListener in run_benchmark_gaddpg_hold.py with modification details::

def _point_state_to_state(self, point_state, ef_pose):
point_state, rgb, depth = point_state[0],point_state[1],point_state[2]
rgb=rgb.numpy()[:,:,:3]
depth = depth.numpy()
depth=depth[:,:,np.newaxis]
rgbd=np.concatenate([rgb,depth],axis=2)
rgbd=np.transpose(rgbd, (2,0,1))
rgbd_resized=np.zeros((4,112,112))
for i in range(rgbd.shape[0]):
rgbd_resized[i]=cv2.resize(rgbd[i],(112,112))

    point_state = self._process_pointcloud(point_state, ef_pose)
    # image_state = np.array([])
    image_state=rgbd_resized
    obs = (point_state, image_state)
    return obs

Is there something wrong with the modification codes?
I would like to know how to introduce RGBD image to the policy network for the handover_sim environment.

The problem of robot in simulation

It is a great job. I want to change robot type in simulation. Does the code provide an interface to change the urdf file of the robot?

pybullet installation

Hi,
I encountered a problem when installing packages according to your process.
Do you have any idea about this problem.

Thanks

Having a problem with compiling assets

When I was running ./handover/data/compile_assets.sh, it came out with:

Compiling MANO assets
Traceback (most recent call last):
  File "compile_mano_assets.py", line 390, in <module>
    main()
  File "compile_mano_assets.py", line 157, in main
    left_hand=mano_side == "left", models_dir=models_dir, betas=mano_betas
TypeError: __init__() got an unexpected keyword argument 'betas'

It turns out that in compile_mano_assets.py at line 157, the model calls HandModel45 with argument 'betas'. Why does this happen and how to fix it? Thanks for answering

Full-body articulated model in pybullet

Hi!

Thanks for your great job to set up both a benchmark for handover task and a training environment of learning-based handover policies. My research group have done some work based on handoversim, like GenH2R(https://genh2r.github.io/). Now to further deal with more complex handover tasks which may includes full-body human motion, I wish to involve full-body model (like smpl+h or smplx) in the simulator. I notice that handoversim uses mano-pybullet as an articulated model, but I cannot find any substitutes for smpl currently.

Have you ever thought about full-body models in pybullet? I would appreciate it if you could provide any information or hints.

TypeError:__init__() got an unexpected keyword argument 'betas'

Hi!

I had this problem when I was compiling 'compile_mano_assets.py'. I cannot figure out why. The error is as below.

Compiling MANO assets
Traceback (most recent call last):
File "compile_mano_assets.py", line 391, in
main()
File "compile_mano_assets.py", line 157, in main
model = HandModel45(
TypeError: init() got an unexpected keyword argument 'betas'

Looking forward to your reply.

Request for Training Code for Human-Robot Handovers in handover-sim Environment

I am impressed by the capabilities and features of the handover-sim environment.

I kindly request your assistance in accessing the training code for human-robot handovers in the handover-sim environment. If it is possible, I would greatly appreciate it if you could share the code with me or provide any guidance on how I can access it. Any additional documentation, instructions, or insights you can provide would be immensely helpful.

b3Printf: URDF file '/home/user/handover-sim/handover/data/assets/table/table.urdf' not found

Hi, thank you for your great work for human-to-robot handovers.

When I run a handover environment (Ubuntu20.04 python 3.8):
python examples/demo_handover_env.py
SIM.RENDER True

I meet the following error:

pybullet build time: May 20 2022 19:44:17
argv[0]=
startThreads creating 1 threads.
starting thread 0
started thread 0
argc=3
argv[0] = --unused
argv[1] =
argv[2] = --start_demo_name=Physics Server
ExampleBrowserThreadFunc started
X11 functions dynamically loaded using dlopen/dlsym OK!
X11 functions dynamically loaded using dlopen/dlsym OK!
Creating context
Created GL 3.3 context
Direct GLX rendering context obtained
Making context current
GL_VENDOR=NVIDIA Corporation
GL_RENDERER=NVIDIA GeForce RTX 4080/PCIe/SSE2
GL_VERSION=3.3.0 NVIDIA 525.105.17
GL_SHADING_LANGUAGE_VERSION=3.30 NVIDIA via Cg compiler
pthread_getconcurrency()=0
Version = 3.3.0 NVIDIA 525.105.17
Vendor = NVIDIA Corporation
Renderer = NVIDIA GeForce RTX 4080/PCIe/SSE2
b3Printf: Selected demo: Physics Server
startThreads creating 1 threads.
starting thread 0
started thread 0
MotionThreadFunc thread started
b3Printf: b3Warning[examples/Importers/ImportURDFDemo/BulletUrdfImporter.cpp,152]:

b3Printf: URDF file '/home/hwpeng/handover-sim/handover/data/assets/table/table.urdf' not found

Traceback (most recent call last):
File "examples/demo_handover_env.py", line 27, in
main()
File "examples/demo_handover_env.py", line 19, in main
env.reset(scene_id=scene_id)
File "/home/user/.conda/envs/handover_sim/lib/python3.8/site-packages/gym/wrappers/order_enforcing.py", line 18, in reset
return self.env.reset(**kwargs)
File "/home/user/.conda/envs/handover_sim/lib/python3.8/site-packages/easysim/simulator_env.py", line 48, in reset
self._simulator.reset(env_ids)
File "/home/user/.conda/envs/handover_sim/lib/python3.8/site-packages/easysim/simulators/bullet.py", line 98, in reset
self._load_body(body)
File "/home/user/.conda/envs/handover_sim/lib/python3.8/site-packages/easysim/simulators/bullet.py", line 178, in _load_body
self._body_ids[body.name] = self._p.loadURDF(body.urdf_file, **kwargs)
pybullet.error: Cannot load URDF file.
ven = NVIDIA Corporation
numActiveThreads = 0
stopping threads
Thread with taskId 0 exiting
Thread TERMINATED
destroy semaphore
semaphore destroyed
destroy main semaphore
main semaphore destroyed
finished
numActiveThreads = 0
btShutDownExampleBrowser stopping threads
Thread with taskId 0 exiting
Thread TERMINATED
destroy semaphore
semaphore destroyed
destroy main semaphore
main semaphore destroyed

Cannot access this repository

Hi,
I hope you are doing good.

I am trying to clone this repository (handoversim) into my local machine using the procedure mentioned in the README of this repo with this command (git clone --recursive [email protected]:NVlabs/handover-sim.git). The problem that my machine raise is :
""
[email protected]: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
""

Can you please fix it as soon as possible?

The problem of ./handover/data/compile_assets.sh

Traceback (most recent call last):
File "compile_mano_assets.py", line 390, in
main()
File "compile_mano_assets.py", line 156, in main
model = HandModel45(
TypeError: init() got an unexpected keyword argument 'betas'

FPS problem

Hi, I'm wondering why is _TIME_STEP_RAW in dex_ycb set to 0.04, in the paper, it says the dataset was collected at 30 fps.

Having problem when installing OMG-Planner

Hi,
I hope you are doing good.

When installing OMG-Planner following "example/README.md", I am trying to execute """cmake ..
-DPYTHON_EXECUTABLE=$VIRTUAL_ENV/bin/python
-DCMAKE_PREFIX_PATH=$( cd ../../orocos_kdl/release && pwd )"""
in terminal and I have modified "$VIRTUAL_ENV". But two errors occur:

1. CMake Error at CMakeLists.txt:18 (find_package):
   find_package called with invalid argument "."

2. CMake Error at cmake/FindSIP.cmake:63 (MESSAGE):
   Could not find SIP
   Call Stack (most recent call first):
   CMakeLists.txt:21 (find_package)

Could you please fix it as soon as possible?

Some errors when compile assests

Hi,
I hope you are doing good.

In step 4 of the Installation, I used option 1 successfully download cased dataset. But in step 5, I am trying to compile assets and when executing "./handover/data/compile_assets. sh" in terminal, the first error is reported:
"""Compiling MANO assets
Traceback (most recent call last):
File "compile_mano_assets.py", line 390, in
main()
File "compile_mano_assets.py", line 156, in main
model = HandModel45(
TypeError: init() got an unexpected keyword argument 'betas'"""

To solve this problem，I directly delete the third parameter of """model = HandModel45(left_hand=mano_side == "left", models_dir=models_dir, betas=mano_betas)""", which is on line 156 in file "compile_mano_assets.py. "

But I am not sure if this is correct ,or it may trigger other errors when running project afterwards.

After solving this problem temporarily and execute “./handover/data/compile_assets.sh” again，another error occurs：
File "/home/handover-sim/mano_pybullet/mano_pybullet/hand_model.py", line 29, in init
super().init(left_hand=left_hand, models_dir=models_dir)
File "/home/lhandover-sim/mano_pybullet/mano_pybullet/mano_model.py", line 29, in init
self._model = self._load(os.path.join(models_dir, fname))
File "/home/handover-sim/mano_pybullet/mano_pybullet/mano_model.py", line 46, in _load
return pickle.load(pick_file, encoding='latin1')
File "/home/anaconda3/envs/handover-sim/lib/python3.8/site-packages/chumpy/init.py", line 11, in
from numpy import bool, int, float, complex, object, unicode, str, nan, inf
ImportError: cannot import name 'bool' from 'numpy'

Could you please fix these problems as soon as possible?

How to replace the Franka Emika Panda robot in the simulator with the Universal Robots' UR5e or UR3e.

I am particularly interested in understanding how to replace the Franka Emika Panda robot in the simulator with the Universal Robots' UR5e or UR3e.

Could you kindly provide guidances or instructions on how I can modify the existing source code to accommodate the Universal Robots' UR5e or UR3e?

Looking forward to hearing from you.

How to improve the GA-DDPG model performance on the "handover_sim" testing environment

I would like to know why the GA-DDPG model trained using the code from the GitHub (https://github.com/liruiw/GA-DDPG) achieves an 87% success rate on the YCB database. However, when I test the same trained model in the "handover_sim" environment, the success rate is only 6.25%, which is significantly worse than the results reported in the paper. I'm wondering if there could be a misalignment between the environment or settings in the GA-DDPG source code and the "handover_sim" testing environment. Is it possible that some additional adjustments to the environment settings are needed to improve the model's performance in the "handover_sim" environment?

The first result below is from my own retrained GA-DDPG model, and the second result is from loading the trained model and testing it in the "handover_sim" environment.

Training code following the GitHub (https://github.com/liruiw/GA-DDPG) setting:
python -m core.train_online --save_model --config_file td3_critic_aux_policy_aux.yaml --policy DDPG --log --fix_output_time ddpg_model_233_1000000_GADDPG --seed 233

Testing code following the GitHub setting:
Testing on YCB objects bash ./experiments/scripts/test_ycb.sh demo_model

---

Test Time: 08_08_2023_11:17:17 Data Root: data/scenes/data_5w.npz Model: demo_model
Script: td3_critic_aux_policy_aux.yaml Index: ycb_large.json
Num of Objs: 9 Num of Runs: 3
Policy: DDPG Model Path: output/demo_model Step: 300000
Test Episodes: 270.0 Avg. Length: 25.815 Index: scene_0-scene_164
Avg. Performance: (Return: 0.870 +- 0.02778) (Success: 0.870 +- 0.02778)
+---------------------+---------+-----------+
| object name | count | success |
|---------------------+---------+-----------|
| 003_cracker_box | 30 | 26 |
| 004_sugar_box | 30 | 23 |
| 005_tomato_soup_can | 30 | 28 |
| 006_mustard_bottle | 30 | 30 |
| 010_potted_meat_can | 30 | 23 |
| 021_bleach_cleanser | 30 | 26 |
| 024_bowl | 30 | 30 |
| 025_mug | 30 | 23 |
| 061_foam_brick | 30 | 26 |
+---------------------+---------+-----------+

run for "GA-DDPG hold" on the test split of s0 with:

GADDPG_DIR=GA-DDPG CUDA_VISIBLE_DEVICES=0 python examples/run_benchmark_gaddpg_hold.py
SIM.RENDER True
ENV.ID HandoverHandCameraPointStateEnv-v1
BENCHMARK.SETUP s0

pybullet build time: May 20 2022 19:44:17
2023-08-07 15:30:16: Running evaluation for results/2023-08-07_13-51-23_ga-ddpg-hold_s0_test
2023-08-07 15:30:16: Evaluation results:
| success rate | mean accum time (s) | failure (%) |

(%)	exec	plan	total	hand contact	object drop	timeout
6.25 ( 9/144)	7.390	0.261	7.651	0.69 ( 1/144)	13.19 ( 19/144)	79.86 (115/144)
2023-08-07 15:30:16: Printing scene ids
2023-08-07 15:30:16: Success (9 scenes):

---

5 8 16 20 25 30 37 55 109

---

2023-08-07 15:30:16: Failure - hand contact (1 scenes):

11

2023-08-07 15:30:16: Failure - object drop (19 scenes):

---

9 12 21 23 24 27 33 36 45 54 60 66 67 68 108 117 121 135 136

---

2023-08-07 15:30:17: Failure - timeout (115 scenes):

---

0 1 2 3 4 6 7 10 13 14 15 17 18 19 22 26 28 29 31 32
34 35 38 39 40 41 42 43 44 46 47 48 49 50 51 52 53 56 57 58
59 61 62 63 64 65 69 70 71 72 73 74 75 76 77 78 79 80 81 82
83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102
103 104 105 106 107 110 111 112 113 114 115 116 118 119 120 122 123 124 125 126
127 128 129 130 131 132 133 134 137 138 139 140 141 142 143

---

2023-08-07 15:30:17: Evaluation complete.

Plan to release IsaacGym version's hanover-sim environment

Hi,

Thanks for building such a nice simulator for handover research.
I read from the paper that you have the IsaacGym version, but when I followed the instructions from Easysim to modify parameters in config files and tried to use IsaacGym, the physical properties of hands, robots and objects seemed to be very wrong. So could you provide an official config file for the IsaacGym environment?