moabitcoin / cherry-pytorch Goto Github PK

As default pre-processing

Adapt agents to the following API proposal

Implement A2C for Atari / Doom games. Inspiration here and here

Currently we don't have push rights to moabitcoin/cherry-torch-<cpu/gpu>. Its recommended to build from source. We want to support Docker pull/run

Currently we use scikit video to write env states to video for visualising the results. We want to move to gym.wrappers.Monitor to do the same.

Sample code from here

• We could not re-produce doom results after porting learnings from Atari
• This bug fix was due to BN layers in Doom which need RMSprop as the optimiser

We want to port learnings from Atari + DQN into Agent of Doom

• Optimized replay buffer
• Lazy loading of tensors into GPU

Currently all agents are training with logging.INFO logging enabled. We'd like to expose user configured logging level enabled from CLI

  parser = argparse.ArgumentParser('Train an RL Agent to'
                                   ' play Atari Game (DQN)')
  parser.add_argument('-x', dest='config_file', type=str,
                      help='Config for the Atari env/agent', required=True)
  parser.add_argument('-d', dest='device', choices=['gpu', 'cpu'],
                      help='Device to run the train/test', default='gpu')
  parser.add_argument('-v', dest='verbosity', choices=['info', 'debug', 'warning, 
'error'],
                      help='Logging verbosity level ', default='info')

Using instructions here. Modify DoomEnvironment to mimic OpenAI gym wrapper

Import learning for Policy Gradients & Actor Critic for ALE (Arcade Learning Environment)

Currently we don't track code version with the model version this can lead to inability to re-produce results.

• Use git python package to get commit tag.
• Add git commit tag as a prefix when saving model files
• Add git commit tag to logs
• Add git commit tags to yaml config files at exp_dir

Currently DoomEnvironment gets default setting from the *.cfg/*.wad files. The default rewards from the DoomGame do not account for any change to the reward based on any changes to GameVariable.

We would like the DoomEnvironment to be configurable via config yaml file to account for changes to rewards based on available GameVariable.

F.ex include GameVariable.AMMO in basic.cfg to budget weapon fire. and GameVariable.HEALTH for health_gathering.cfg to incentivise gathering of medkits

Currently we only support using cherry within a conda 🐍 environment. We'd like to add support for building/running docker image. Following functionalities are handy

1. Building docker image for cherry
2. Publishing docker image on Docker Hub

Sample setup from here

We need explanation on the design principles and documentations of agents/model architectures/environment

We want to port the following changes from DQN into DDQN

• Optimised memory buffer for replay
• Using wrapper_deepmind for Atari games

Bring in learnings from Atari into Doom for faster learning

We currently train Atari models with DQN + DDQN. VPG (Vanilla Policy Gradient) have shown to be a better structured agent (as tested on Control problems / Health gathering in Doom).

Generate model + hyper params which can solve Breakout Atari retro game

We want to support Trusted Region Policy Optimisation and Proximal Policy Optimisation

Currently at playtime the gameplay + actions are written to a video file. We'd like an option for viewing the current gameplay rendered in realtime for debugging.

Use PyTorch dataloader class to read from buffer/replay than running sequential over the batch

Implement ideas from here

Currently we keep 2 * 4 frame states in buffer this can be reduced to 5 with first 4 frames the current state and last 4 the next state

Use TensorboardX for monitoring training / validation session

Adapt DoomEnviroment and yaml config files to solve health_gathering scenario.

Since ours is an open source project we'd like to use Travis CI for building / testing a PR/commits. This would also push the built images to Docker Hub for distribution.

Sample usage from here

Currently the codebase is organised as follows

├── ddqn
│   ├── atari
│   └── doom
├── dqn
│   ├── atari
│   └── doom
├── policy_gradients
│   ├── atari
│   ├── classic_control
│   └── doom
├── q-learning
└── utils

Rather than running via an entry scripts as we do currently. We'd like a cli : bin/cherry
We'd to like to re-organise as follow

├── agents
│   ├── dqn
│   └── ddqn
│   ├── q_learning
│   └── policy_gradient
├── envs
│   ├── atari
│   └── doom
│   └── classic_control
├── configs
│   ├── doom.yaml
│   └── atari.yaml
├── utils
└── bin/cherry

And run as follows

bin/cherry train -cfg configs/doom.yaml
bin/cherry play -cfg configs/doom.yaml

Provide convenient download link for pre-trained models with performance/evaluation guarantees.

As default Frame Queue

Currently we include null_state to infer terminal or game over state. This can be factored out by using (1 - done) * gamma * Q

DQN & DDQN don't perform well for large action space problems like deadly_corridor & health_gathering. For this we want to explore

• Policy gradients
• Actor Critic models