uribalb / osim-rl

Reinforcement learning environments with musculoskeletal models

NeurIPS 2019: Learn to Move - Walk Around

This repository contains software required for participation in the NeurIPS 2019 Challenge: Learn to Move - Walk Around. See more details about the challenge here. See full documentation of our reinforcement learning environment here. In this document we will give very basic steps to get you set up for the challenge!

Your task is to develop a controller for a physiologically plausible 3D human model to walk or run following velocity commands with minimum effort. You are provided with a human musculoskeletal model and a physics-based simulation environment, OpenSim. There will be three tracks:

1. Best performance
2. Novel ML solution
3. Novel biomechanical solution, where all the winners of each track will be awarded.

To model physics and biomechanics we use OpenSim - a biomechanical physics environment for musculoskeletal simulations.

What's new compared to NIPS 2017: Learning to run?

We took into account comments from the last challenge and there are several changes:

• You can use experimental data (to greatly speed up learning process)
• We released the 3rd dimensions (the model can fall sideways)
• We added a prosthetic leg -- the goal is to solve a medical challenge on modeling how walking will change after getting a prosthesis. Your work can speed up design, prototying, or tuning prosthetics!

You haven't heard of NIPS 2017: Learning to run? Watch this video!

Getting started

Anaconda is required to run our simulations. Anaconda will create a virtual environment with all the necessary libraries, to avoid conflicts with libraries in your operating system. You can get anaconda from here https://docs.anaconda.com/anaconda/install/. In the following instructions we assume that Anaconda is successfully installed.

For the challenge we prepared OpenSim binaries as a conda environment to make the installation straightforward

We support Windows, Linux, and Mac OSX (all in 64-bit). To install our simulator, you first need to create a conda environment with the OpenSim package.

On Windows, open a command prompt and type:

conda create -n opensim-rl -c kidzik -c conda-forge opensim python=3.6.1
activate opensim-rl
pip install osim-rl

On Linux/OSX, run:

conda create -n opensim-rl -c kidzik -c conda-forge opensim python=3.6.1
source activate opensim-rl
pip install osim-rl

These commands will create a virtual environment on your computer with the necessary simulation libraries installed. If the command python -c "import opensim" runs smoothly, you are done! Otherwise, please refer to our FAQ section.

Note that source activate opensim-rl activates the anaconda virtual environment. You need to type it every time you open a new terminal.

Basic usage

To execute 200 iterations of the simulation enter the python interpreter and run the following:

from osim.env import L2M2019Env

env = L2M2019Env(visualize=True)
observation = env.reset()
for i in range(200):
    observation, reward, done, info = env.step(env.action_space.sample())

The function env.action_space.sample() returns a random vector for muscle activations, so, in this example, muscles are activated randomly (red indicates an active muscle and blue an inactive muscle). Clearly with this technique we won't go too far.

Your goal is to construct a controller, i.e. a function from the state space (current positions, velocities and accelerations of joints) to action space (muscle excitations), that will enable to model to travel as far as possible in a fixed amount of time. Suppose you trained a neural network mapping observations (the current state of the model) to actions (muscle excitations), i.e. you have a function action = my_controller(observation), then

# ...
total_reward = 0.0
for i in range(200):
    # make a step given by the controller and record the state and the reward
    observation, reward, done, info = env.step(my_controller(observation))
    total_reward += reward
    if done:
        break

# Your reward is
print("Total reward %f" % total_reward)

You can find details about the observation object here.

Submission

• Option 1: submit solution in docker container
• Option 2: run controller on server environment: ./examples/submission.py

In order to make a submission to AIcrowd, please refer to this page

Rules

Organizers reserve the right to modify challenge rules as required.

Read more in the official documentation

• Osim-rl interface
• How to train a model?
• More on training models
• Experimental data
• Physics underlying the model
• Frequently Asked Questions
• Citing and credits
• OpenSim documentation

Contributions of participants

• Understanding the Challenge - Great materials from @seungjaeryanlee on how to start

Partners