cpezzato / hab-mobile-manipulation

Mobile manipulation in Habitat

Mobile Manipulation for the Home Assistant Benchmark (HAB)

This is modified version of the PyTorch implementation of:

Multi-skill Mobile Manipulation for Object Rearrangement
Jiayuan Gu, Devendra Singh Chaplot, Hao Su, Jitendra Malik
UC San Diego, Meta AI Research, UC Berkeley

Project website: https://sites.google.com/view/hab-m3

set_table_teaser.mp4

Table of Contents

• Installation
• Interactive play example
• Evaluation
  • Evaluate a sub-task
  • Evaluate a HAB (Home Assistant Benchmark) task
• Training
• Acknowledgments

Installation

Prepare the conda environment

conda create -n habitat-env python=3.10 -y
conda activate habitat-env
conda install cmake=3.14.0 patchelf ninja -y

Install everything

This will install habitat sim, habitat lab, and this package. It will also download the necessary datasets.

cd hab-mobile-manipulation
./install.sh

Interactive play example

Interactively play the task with the default config:

conda activate habitat-env
python habitat_extensions/tasks/rearrange/play.py

Use i/j/k/l to move the robot end-effector, and w/a/s/d to move the robot base. Use f/g to grasp or release an object.

---

Use a specific task config:

python habitat_extensions/tasks/rearrange/play.py  --cfg configs/rearrange/tasks/pick_v1.yaml

Or use a specific RL config:

python habitat_extensions/tasks/rearrange/play.py  --cfg configs/rearrange/skills/tidy_house/pick_v1_joint_SCR.yaml

Evaluation

Pretrained skills can be downloaded via:

gdown https://drive.google.com/drive/folders/1u7DAd25PE818wjg-MxDKJ7y5n8GQtfrz -O data/results/rearrange/skills --folder

If this fails, you can manually add to the data/ folder the pretrained models from here. Simply download and unzip the results/ folder into data/.

Evaluate a sub-task

# Evaluate the latest checkpoint of a skill saved at "data/results/rearrange/skills/tidy_house/pick_v1_joint_SCR"
python mobile_manipulation/run_ppo.py --cfg configs/rearrange/skills/tidy_house/pick_v1_joint_SCR.yaml --run-type eval
# Evaluate the latest checkpoint of a skill saved at "data/results/rearrange/skills/tidy_house/pick_v1_joint_SCR/seed=100"
python mobile_manipulation/run_ppo.py --cfg configs/rearrange/skills/tidy_house/pick_v1_joint_SCR.yaml --run-type eval --run-type eval PREFIX seed=100

Evaluate a HAB (Home Assistant Benchmark) task

# Interactively visualize results
python mobile_manipulation/eval_composite.py --cfg configs/rearrange/composite/tidy_house/mr.yaml --viewer --render-info
# Save results
# "--no-rgb" can be passed to the script to accelerate if rgb is not used.
python mobile_manipulation/eval_composite.py --cfg configs/rearrange/composite/tidy_house/mr.yaml --save-log
# Save videos
python mobile_manipulation/eval_composite.py --cfg configs/rearrange/composite/tidy_house/mr.yaml --save-video all

---

We provide a script to evaluate a HAB task for multiple runs (different training and evaluation seeds).

BG=1 bash scripts/eval_composite.sh configs/rearrange/composite/tidy_house/mr.yaml

To collect evaluation results of multiple runs:

# By default, there will also be a timestamp in the path
python scripts/collect_results.py -d data/results/rearrange/composite/tidy_house/mr

Training

Train an individual skill (e.g., Pick for TidyHouse):

# The result will be saved at "data/results/rearrange/skills/tidy_house/pick_v1_joint_SCR".
python mobile_manipulation/run_ppo.py --cfg configs/rearrange/skills/tidy_house/pick_v1_joint_SCR.yaml --run-type train
# Specify a prefix and a random seed for the experiment.
# The result will be saved at "data/results/rearrange/skills/tidy_house/pick_v1_joint_SCR/seed=101" 
python mobile_manipulation/run_ppo.py --cfg configs/rearrange/skills/tidy_house/pick_v1_joint_SCR.yaml --run-type train PREFIX seed=101 TASK_CONFIG.SEED 101

Acknowledgments

Many thanks to Jiayuan Gu, Devendra Singh Chaplot, Hao Su, Jitendra Malik, authors of the original repository.