miroesli / psscscs

Reinforcement Learning Battlesnake

~$ whoami

Hi there 👋,

My name is Michail Roesli and my dream is to provide a meaningful difference to the world through data analytics and machine learning.

👦 About Me

• 🏫 Graduated from the University of Victoria with a focus in Computational Intelligence
• 💻 Learning Rust and Julia on the side
• 🔭 Always on the lookout to learn new things and explore new passion projects
• 🐧 I love Linux and open source projects ❤️

🔗 Contact

• LinkedIn: https://www.linkedin.com/in/miroesli/
• Personnal website: https://miroesli.github.io/

psscscs's Issues

Fix Heroku build requirements package error

       ERROR: tensorboard 1.14.0 has requirement setuptools>=41.0.0, but you'll have setuptools 39.0.1 which is incompatible.

       Installing collected packages: bottle, greenlet, gevent, gunicorn, six, numpy, scipy, keras-preprocessing, h5py, keras-applications, pyyaml, keras, keras-rl, astor, termcolor, wrapt, gast, google-pasta, protobuf, absl-py, grpcio, werkzeug, markdown, tensorboard, tensorflow-estimator, tensorflow

Mess around with algorithm (optimizer functions) parameters to see which models do best

Explain how to run main.py file and how to train models in README

Heroku deployment

If we can run this on a lightweight platform like heroku I think it'd be ideal. See if we can stand this up on Heroku and how it performs

Help me to run the test

for it is easy for a human to read,
I have switched on the state printing in game.py
run this file with command python run_this_test_and_output_to_a_file.py > output.txt
see if the results make sense
a 1-8 number represents a snake body, with value = its id
9 represents a food
feel free to make your own test

you won't be able to see dead snakes' bodies or how it died. In other words, they just disappear one turn before they make the stupid move.

Need to determine which python version is required for running the required tensorflow version

Rewards for DQN

Rewards functionality in the gym
Look in the env/ folder
Instantiate a new class in there
If you win, add one score, if you lose deduct one score

Import array from numpy in the neuralnet.py file (the file which contains the alphazero algorithm)

In the train_alpha() method in the AlphaZero algorithm file (app/algs/neuralnet.py), we are calling the train function on the new_nnet, but we're passing the following parameters: (array(X), array(Y)). However, we are not importing the array function. Simple fix, just import the array function from numpy.

Create MCTS

Create a monte carlo tree search for AlphaZero algorithm

Fix parameter passsing to AlphaGo

The config file isn't being passed in properly to the AlphaGoZero Algorithm, fix it.

Cannot run the `main.py` local snake

Use Rotation and Reflection to Increase Dataset

Use rotation and reflection on the input states to give us 8x more states with 1 state

Change The Compete and Train_Alpha Functions in AlphaZero Algorithm to Be An Instance Method

In app/algs/neuralnet.py, the compete() and train_alpha() methods should be changed so that they are instance methods. This includes changing the first function parameter to be "self". Simple hotfix required. Additionally, this change needs to be refactored throughout the same file, so that when we're calling the compete and train_alpha functions, we're calling it using self.function_name(params).

Store trained neural net in app/train.py in app/models/

In this issue, we need to store our trained, amazing neural network to disk in the app/models/ folder so we can actually use it later on. Everything to do this is more or less already set in place. When we run the train function for our alg, we'll get returned back a AlphaNNet object. These objects already have an instance function called save. The only thing that I think will need to be changed is the save function will need to specify the models folder. So the function will need to be changed from self.nnet.save(name + '.h5') to self.nnet.save('./models/' + name + '.h5') although not a 100% sure about this. Verification is left to the assignee.

Add alg loader

This will replace how train calls the algorithms and how main initializes on deployment.

It will be imported from utils

Big update. I rewrote the battlesnake game!!!

The game.py (which uses snake.py) is a battlesnake game that can be run totally locally, so we can start training on our machines without having to deal with browsers and web-based interfaces. It also uses optimized data structures so it runs much faster (at least in theory; they use the language "go" so I don't know how that is compared to python). I also added a translator to convert the online data to our new data.

Remove NonMLsnake.py model

Refractor Class & File Name "neuralnet" to "AlphaZero" in app/algs/neuralnet.py

This issue relates to refactoring the name from neuralnet to AlphaZero since I think it's a more appropriate name for the class and file.

@Fool-Yang Does this make sense? Or am I misunderstanding the code?

Main.py config default alg used

Allow the use of different neural networks instead of just AlphaNNet through the json files.

Allow real-world training

It might be beneficial to be able to pit our snake against others by including training functionality in our API

Do some preprocessing for DQN algorithm

This may include:

• Change input state for DQN algorithm to be similar to how Yung modified the input state so we can see snake length, snake direction etc.

Add functionality to run a model in the main.py file

• move function currently reads the default config, but does not run the specified model to obtain the direction for next move

Create a Loss Function for AlphaZero Training Function

The train() function for the AlphaZero algorithm needs a loss function.

Create DQN algorithm

• consider implementing with openai's battlesnake env that amazon created. A tutorial for using the environment can be found in the following links:
  • awslabs-battlesnake-ai repo
  • creating your own environment
  • using dqn with an environment
• compare performances with existing environment and also compare algorithm performance with alphazero

Latest update notes and what we should do next

1. a bat script is added to the root dir along with the game engine. If you uses windows, then double click the .bat file will run my snake locally and open the browser.

2. two new folders added in the app folder. algorithms and models

3. TEMP.py renamed to preprocess and is now in the algorithms folder

Add logging functionality to alphazero to store in data folder

Improve training efficiency

In order to drastically speed up training times we should eventually artificially increase our training dataset sizes by reflecting the game board, this essentially halves the possible states we need to analyze. See here for an explanation https://nanonets.com/blog/data-augmentation-how-to-use-deep-learning-when-you-have-limited-data-part-2/ . If we include rotation in this mix we can cut possible states down to 1/8th of the original size

The algorithm is almost done.

I used the algorithm Alphago Zero used to train the network.
nature24270.pdf
Simple Alpha Zero.pdf

Fix Relative Import Issue

Problem with main.py where running main.py results in an error because we were using relative importing because the linter was complaining.

Fix trainer selector

Change the train selection to choose the one specified by the input JSON, otherwise it should choose template. It should not ever create template trainer before selecting another one.

see lines 49 to 51 in train.py.

Essentially, change this to what it was before.

Complete winner_id if statement in alphazero