cbfinn / maml_rl

Code for RL experiments in "Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks"

Model-Agnostic Meta-Learning

This repo contains code accompaning the paper, Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (Finn et al., ICML 2017). It includes code for running the few-shot reinforcement learning experiments.

For the experiments in the supervised domain, see this codebase.

Dependencies

This code is based off of the rllab code repository and can be installed in the same way (see below). This codebase is not necessarily backwards compatible with rllab.

The MAML code uses the TensorFlow rllab version, so be sure to install TensorFlow v1.0+.

Usage

Scripts for running the experiments found in the paper are located in maml_examples/.

The pointmass environment is located in maml_examples/ whereas the MuJoCo environments are located in rllab/envs/mujoco/.

Speed of Code

One current limitation of the code is that it is particularly slow. We welcome contributions to speed it up. We expect the biggest speed improvements to come from better parallelization of sampling and meta-learning graph computation.

Contact

To ask questions or report issues, please open an issue on the issues tracker.

rllab

rllab is a framework for developing and evaluating reinforcement learning algorithms. It includes a wide range of continuous control tasks plus implementations of the following algorithms:

• REINFORCE
• Truncated Natural Policy Gradient
• Reward-Weighted Regression
• Relative Entropy Policy Search
• Trust Region Policy Optimization
• Cross Entropy Method
• Covariance Matrix Adaption Evolution Strategy
• Deep Deterministic Policy Gradient

rllab is fully compatible with OpenAI Gym. See here for instructions and examples.

rllab only officially supports Python 3.5+. For an older snapshot of rllab sitting on Python 2, please use the py2 branch.

rllab comes with support for running reinforcement learning experiments on an EC2 cluster, and tools for visualizing the results. See the documentation for details.

The main modules use Theano as the underlying framework, and we have support for TensorFlow under sandbox/rocky/tf.

Documentation

Documentation is available online: https://rllab.readthedocs.org/en/latest/.

Citing rllab

If you use rllab for academic research, you are highly encouraged to cite the following paper:

• Yan Duan, Xi Chen, Rein Houthooft, John Schulman, Pieter Abbeel. "Benchmarking Deep Reinforcement Learning for Continuous Control". Proceedings of the 33rd International Conference on Machine Learning (ICML), 2016.

Credits

rllab was originally developed by Rocky Duan (UC Berkeley / OpenAI), Peter Chen (UC Berkeley), Rein Houthooft (UC Berkeley / OpenAI), John Schulman (UC Berkeley / OpenAI), and Pieter Abbeel (UC Berkeley / OpenAI). The library is continued to be jointly developed by people at OpenAI and UC Berkeley.

Slides

Slides presented at ICML 2016: https://www.dropbox.com/s/rqtpp1jv2jtzxeg/ICML2016_benchmarking_slides.pdf?dl=0

maml_rl's Issues

MAMLNPO meta-update

How can I get the oracle policy ?

file ../vendor/mujoco/libmujoco131 is missing

when I run test_maml_cheetah.py cause OSerror:
OSError: ../vendor/mujoco/libmujoco131.so: cannot open shared object file: No such file or directory
can you append the need part?

thank you!

Cannot set up anaconda environment

Hi,
I'm trying to set up the environment with Anaconda to run the code.
I'm running: conda env create -f environment.yml and I get error:

Warning: you have pip-installed dependencies in your environment file, but you do not list pip itself as one of your conda dependencies.  Conda may not use the correct pip to install your packages, and they may end up in the wrong place.  Please add an explicit pip dependency.  I'm adding one for you, but still nagging you.
Collecting package metadata (repodata.json): done
Solving environment: failed

ResolvePackageNotFound:
  - joblib==0.9.4
  - numpy==1.10.4
  - python==3.5.2

Do you know how to fix that?

ImportError: No module named 'sandbox.rocky.tf.algos.maml_trpo'

Here is my Error:

Traceback (most recent call last):
  File "maml_examples/cluster_maml_trpo_ant.py", line 1, in <module>
    from sandbox.rocky.tf.algos.maml_trpo import MAMLTRPO
ImportError: No module named 'sandbox.rocky.tf.algos.maml_trpo'

I have installed rllab3 , and set PYTHONPATH properly.
then source activate rllab3 , go to maml_rl directory and run python maml_examples/cluster_maml_trpo_ant.py , and this error occurs.
What's wrong? should I git clone maml_rl into some directory under rllab?

Half cheetah observations with self.get_body_com("torso").flat

Hello! I'm wondering if there are any good reasons for adding self.get_body_com("torso").flat to the observations of half cheetah:

maml_rl/rllab/envs/mujoco/half_cheetah_env_rand_direc.py

Line 53 in 9c8e2eb

self.get_body_com("torso").flat,

It seems to me that it only complicates comparisons between methods which also use gym.
Thank you!

MAML for DDPG

Hi,
I'd like to ask few important questions,

• is it possible to implement MAML with DDPG like other implemented MAML-RL algorithms?
• if so, what are the right way to do that? and how much we expect the DDPG performance will improve after adding MAML?

regards,

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.binstar.org/tlatorre/noarch/repodata.json>

Hi, I run ./scripts/setup_linux.sh and get the following error:

Creating conda environment...
Using Anaconda API: https://api.anaconda.org
Solving environment: failed

CondaHTTPError: HTTP 000 CONNECTION FAILED for url <https://conda.binstar.org/tlatorre/noarch/repodata.json>
Elapsed: -

An HTTP error occurred when trying to retrieve this URL.
HTTP errors are often intermittent, and a simple retry will get you on your way.
SSLError(MaxRetryError('HTTPSConnectionPool(host=\'conda.binstar.org\', port=443): Max retries exceeded with url: /tlatorre/noarch/repodata.json (Caused by SSLError(SSLError("bad handshake: Error([(\'SSL routines\', \'ssl3_get_server_certificate\', \'certificate verify failed\')],)",),))',),)

I have tried many times and the error always occurs. I wonder whether the version you used can not work now ? or other problems.
Thanks.

Bug when init VariantGenerator

Hi, I try to run the code. Last day, I success. But today I meet a strange problem.
I don't modify any code, just use setup_mujoco.sh script from
to setup the mujoco env for running cluster_trpo_cheetah.py:
For bug information see:

It seems that the VG class init with 4 arguments but default init is 2. But I check the class VariantGenerator, it's init with no arguments. So with this bug information I can't locate where the problem is. Have you ever meet this problem.

Where to find the data file？

Can not find where to fit the linear baseline

Hello,

I'm trying to implement the maml_rl for PyTorch, while I couldn't find where you fit the baseline.
Can you help me point it out?

Installation/replicating point mass experiments

Hi, I've been trying to replicate some of your results and compare against Reptile, for the reinforcement learning case. Specifically, I'm trying to run the point mass experiments.

After installing everything and sourcing the conda env rllab3 and running python maml_examples/maml_trpo_point.py, it complains about Theano and Lasagne not being installed.

From the README, it seems like this was built on the Tensorflow version of rllab. Does it still need Theano or am I doing something wrong?

PS: Thanks for open-sourcing your work!

AttributeError: function 'mjv_makeObjects' not found

Hello,
I am trying to set up Mujoco for one of my projects. I installed all the dependencies and used the appropriate license file. When I am trying to run test_maml_ant.py, its giving me the following error

func = self._FuncPtr((name_or_ordinal, self))
AttributeError: function 'mjv_makeObjects' not found

in the file mjlib.py. Does anyone know if it is the installation issue or am I missing something? The examples provided by the mujoco_py repositories are working without any error though.