microsoft / bonsai-gym

Bonsai Gym

A python library for integrating Bonsai brains with Open AI Gym environments.

Bonsai Gym

A python library for integrating Bonsai BRAIN with Open AI Gym environments.

Installation

One of the requirements bonsai-common is not located on pypi and must be installed from the github repo

$ pip install git+https://github.com/microsoft/bonsai-common

Install the latest bonsai-gym from github.

NOTE - The bonsai-gym located on pypi is an older release that we no longer support. We have no plans to publish this to pypi

$ pip install git+https://github.com/microsoft/bonsai-gym

Usage

Once installed, import bonsai_gym in order to access base class GymSimulator3, which implements all of the environment-independent Bonsai SDK integrations necessary to train a Bonsai BRAIN to play an OpenAI Gym simulator.

See CartPole sample

    import gym

    from bonsai_gym import GymSimulator3

    class CartPole(GymSimulator3):
        # Environment name, from openai-gym
        environment_name = "CartPole-v0"

        # Simulator name from Inkling
        simulator_name = "CartpoleSimulator"

        # convert openai gym observation to our state type
        def gym_to_state(self, observation):
            state = {
                "position": observation[0],
                "velocity": observation[1],
                "angle": observation[2],
                "rotation": observation[3],
            }
            return state

        # convert our action type into openai gym action
        def action_to_gym(self, action):
            return action["command"]


    if __name__ == "__main__":
        # create a brain, openai-gym environment, and simulator
        config = BonsaiClientConfig(argv=sys.argv)
        sim = CartPole(config)
        sim.run_gym()

Running Simulator

Simulators need two environment variables set to be able to attach to the platform.

The first is SIM_ACCESS_KEY. You can create one from the Account Settings page. You have one chance to copy the key once it has been created. Make sure you don't enter the ID.

The second is SIM_WORKSPACE. You can find this in the URL after /workspaces/ once you are logged in to the platform.

There is also an optional SIM_API_HOST key, but if it is not set it will default to https://api.bons.ai.

If you're launching your simulator from the command line, make sure that you have these two environment variables set. If you like, you could use the following example script:

export SIM_WORKSPACE=<your-workspace-id>
export SIM_ACCESS_KEY=<your-access-key>
python3 my_sim.py

Rewards and Terminals

The GymSimulator3 class automatically appends the gym reward and gym terminal to the state extracted from the environment with the keys named _gym_reward and _gym_terminal respectively. You can use these rewards and terminals in your Inkling definition using Inkling functions. See the example Inklings in the samples directory to see a standard implementation.

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

bonsai-gym's Issues

All samples have inkling that are missing rewards or goal functions

Suggestion: Can we make it easier to use exported brains with bonsai-gym?

    def run_exported_brain(self, num_episodes: int = 500, num_iterations: int = 200, render: bool = False, exported_brain_url: str = "http://localhost:5000"):

        for episode in range(num_episodes):
            terminal = False
            iteration = 0
            while not terminal:
                iteration += 1
                random_action = self._env.action_space.sample()
                _, reward, terminal, _ = self._env.step(random_action)
                prediction_endpoint = f"{exported_brain_url}/v1/prediction"
                response = requests.get(prediction_endpoint, json=state)
                action = response.json()
                log.info(
                    f"Episode: #{episode}, iteration: #{iteration}, reward: {reward}, Terminal: {terminal}"
                )
                if iteration > num_iterations - 1: terminal = True
                if not render or self._headless:
                    if "human" in self._env.metadata["render.modes"]:
                        self._env.render()