zoruasama / madmario

Interactive tutorial to build a learning Mario, for first-time RL learners

MadMario

PyTorch official tutorial to build an AI-powered Mario.

Set Up

1. Install conda

2. Install dependencies with environment.yml

   conda env create -f environment.yml
   

   Check the new environment mario is created successfully.

3. Activate mario enviroment

   conda activate myenv
   

Running

To start the learning process for Mario,

python main.py

This starts the double Q-learning and logs key training metrics to checkpoints. In addition, a copy of MarioNet and current exploration rate will be saved.

GPU will automatically be used if available. Training time is around 80 hours on CPU and 20 hours on GPU.

To evaluate a trained Mario,

python replay.py

This visualizes Mario playing the game in a window. Performance metrics will be logged to a new folder under checkpoints. Change the load_dir, e.g. checkpoints/2020-06-06T22-00-00, in Mario.load() to check a specific timestamp.

Project Structure

main.py Main loop between Environment and Mario

agent.py Define how the agent collects experiences, makes actions given observations and updates the action policy.

wrappers.py Environment pre-processing logics, including observation resizing, rgb to grayscale, etc.

neural.py Define Q-value estimators backed by a convolution neural network.

metrics.py Define a MetricLogger that helps track training/evaluation performance.

tutorial.ipynb Interactive tutorial with extensive explanation and feedback. Run it on Google Colab.

Key Metrics

• Episode: current episode
• Step: total number of steps Mario played
• Epsilon: current exploration rate
• MeanReward: moving average of episode reward in past 100 episodes
• MeanLength: moving average of episode length in past 100 episodes
• MeanLoss: moving average of step loss in past 100 episodes
• MeanQValue: moving average of step Q value (predicted) in past 100 episodes

Pre-trained

Checkpoint for a trained Mario: https://drive.google.com/file/d/1RRwhSMUrpBBRyAsfHLPGt1rlYFoiuus2/view?usp=sharing

Resources

Deep Reinforcement Learning with Double Q-learning, Hado V. Hasselt et al, NIPS 2015: https://arxiv.org/abs/1509.06461

OpenAI Spinning Up tutorial: https://spinningup.openai.com/en/latest/

Reinforcement Learning: An Introduction, Richard S. Sutton et al. https://web.stanford.edu/class/psych209/Readings/SuttonBartoIPRLBook2ndEd.pdf

super-mario-reinforcement-learning, GitHub: https://github.com/sebastianheinz/super-mario-reinforcement-learning

Deep Reinforcement Learning Doesn't Work Yet: https://www.alexirpan.com/2018/02/14/rl-hard.html