sparisi / pvr_habitat

Pre-Trained Visual Representations for Control

This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

---

Implementation of PVR for Control, as presented in The (Un)Surprising Effectiveness of Pre-Trained Vision Models for Control.

The main branch reproduces the results presented in the paper.
The distributed branch uses multiprocessing and distributed training for much faster learning. It also has wrappers for other environments, including Adroit and DMC. Results are slightly different, but we encourage to use it if you would like to build upon our paper.

Codebase Installation

conda create -n pvr_habitat python=3.8
conda activate pvr_habitat
git clone [email protected]:sparisi/pvr_habitat.git
cd pvr_habitat
pip install -r requirements.txt

Habitat Installation

• Clone https://github.com/sparisi/habitat-lab and do a full install with habitat_baselines.

The main differences between this and the original Habitat repository are:

1. STOP action removed,
2. Bugfix where the agent is placed slightly above the ground, and therefore the terminal goal condition is never triggered.

• Download and extract Replica scenes in the root folder of pvr_habitat.

WARNING! The dataset is very large!

sudo apt-get install pigz
git clone https://github.com/facebookresearch/Replica-Dataset.git
cd Replica-Dataset
./download.sh replica-path

If you have already downloaded it somewhere else, just make a symbolic link

ln -s path/to/Replica-Dataset Replica-Dataset

How to Run Experiments

There are three main scripts to run behavioral cloning:

• main_bc_1.py loads raw trajectories saved as pickles, passes observations (images) through the embedding, and then learns on the embedded observations.
• main_bc_2.py directly loads embedded observations that have already been passed through the embedding, in order to save time.
• main_bc_finetune.py is used to finetune the random PVR.

For more details on how to generate trajectories and pickles, see the README in the behavioral_cloning folder.

Pre-trained models can be downloaded here.

pvr_habitat's Issues

Error while running pip install -r requirements.txt

Running command git clone --filter=blob:none --quiet https://github.com/facebookresearch/detectron2.git /tmp/pip-req-build-gyjrz5o0
Resolved https://github.com/facebookresearch/detectron2.git to commit 864913f0e57e87a75c8cc0c7d79ecbd774fc669b
error: subprocess-exited-with-error

× python setup.py egg_info did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.
Preparing metadata (setup.py) ... error
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Question regarding success rates

Hello @sparisi ,

I am currently working on replicating results from your paper with the intention to extend the work.
I used the 'main' branch and downloaded the pretrained models provided using the github link. I was able to replicate the ground truth results for the Habitat dataset but having little trouble replicating results for - resnet50 models pretrained on ImageNet for embedding name - resnet50_l3 and resnet50_l4. I am running following commands:

Generated Optimal trajectories for 5 scenes

python behavioral_cloning/save_opt_trajectories.py --env=HabitatImageNav-apartment_0
python behavioral_cloning/save_opt_trajectories.py --env=HabitatImageNav-frl_apartment_0
python behavioral_cloning/save_opt_trajectories.py --env=HabitatImageNav-room_0
python behavioral_cloning/save_opt_trajectories.py --env=HabitatImageNav-hotel_0
python behavioral_cloning/save_opt_trajectories.py --env=HabitatImageNav-office_0

generated desired embeddings

python behavioral_cloning/save_embedded_obs.py --env=HabitatImageNav-apartment_0 --embedding_name=resnet50_l3 --source=pickle
python behavioral_cloning/save_embedded_obs.py --env=HabitatImageNav-frl_apartment_0 --embedding_name=resnet50_l3 --source=pickle
python behavioral_cloning/save_embedded_obs.py --env=HabitatImageNav-room_0 --embedding_name=resnet50_l3 --source=pickle
python behavioral_cloning/save_embedded_obs.py --env=HabitatImageNav-hotel_0 --embedding_name=resnet50_l3 --source=pickle
python behavioral_cloning/save_embedded_obs.py --env=HabitatImageNav-office_0 --embedding_name=resnet50_l3 --source=pickle

running bc files - to train the policy network for 5 seeds

python main_bc_2.py --env=HabitatImageNav-apartment_0,HabitatImageNav-frl_apartment_0,HabitatImageNav-room_0,HabitatImageNav-hotel_0,HabitatImageNav-office_0 --embedding_name=resnet50_l3 --run_id=1

python main_bc_2.py --env=HabitatImageNav-apartment_0,HabitatImageNav-frl_apartment_0,HabitatImageNav-room_0,HabitatImageNav-hotel_0,HabitatImageNav-office_0 --embedding_name=resnet50_l3 --run_id=2

python main_bc_2.py --env=HabitatImageNav-apartment_0,HabitatImageNav-frl_apartment_0,HabitatImageNav-room_0,HabitatImageNav-hotel_0,HabitatImageNav-office_0 --embedding_name=resnet50_l3 --run_id=3

python main_bc_2.py --env=HabitatImageNav-apartment_0,HabitatImageNav-frl_apartment_0,HabitatImageNav-room_0,HabitatImageNav-hotel_0,HabitatImageNav-office_0 --embedding_name=resnet50_l3 --run_id=4

python main_bc_2.py --env=HabitatImageNav-apartment_0,HabitatImageNav-frl_apartment_0,HabitatImageNav-room_0,HabitatImageNav-hotel_0,HabitatImageNav-office_0 --embedding_name=resnet50_l3 --run_id=5

computed the mean across five seeds for the last 6 policy updates as described in the paper - i am not able to get the same value as mentioned. Can you please let me know what I am doing wrong?

###I also tried testing the models by running following command again - model_test.py file

python main_test.py --from_env=HabitatImageNav-apartment_0,HabitatImageNav-frl_apartment_0,HabitatImageNav-room_0,HabitatImageNav-hotel_0,HabitatImageNav-office_0 --embedding_name=resnet50_l3 --to_env=HabitatImageNav-apartment_0

Thank you for the help!

Question: regarding HMS data generation code

Hello @sparisi,

Can you please let me know your timeline for uploading data generation code for HMS dataset? I would like to repeat your experiments for HMS dataset as well was wondering about your timeline to upload the code based on your comment below on the README.

Hand Manipulation Suite: coming soon.
More data will be added in the future.

Thank you,
Manju