joshvarty / haliteio Goto Github PK

Input for ship and spawn

- 64x64 plane for Halite           
- 64x64 plane for current unit      
- 64x64 plane for my other units
- 64x64 plane for my units halite
- 64x64 plane for my shipyard
- 64x64 plane for my dropoff
- 64x64 plane for enemy units
- 64x64 plane for enemy units
- 64x64 plane for enemy units
- 64x64 plane for enemy units halite
- 64x64 plane for enemy units halite
- 64x64 plane for enemy units halite

 -64x64 plane for my total score
- 64x64 plane for enemy score
- 64x64 plane for enemy score
- 64x64 plane for enemy score

- 64x64 plane for number of steps (maybe how many steps are left?)

• Finish removing spawn stuff (go back to fixed schedule)
• Correct MyBot.cpp for new approach
• Visualize current performance
• Remove dropoff action?
• Set up GCloud?

I'm worried about high correlations between rollouts generated from a single game. (Halite is identical, dropoffs identical, ships usually in the same space etc.)

Consider training on every second or third example and just play more games to create enough rollout items.

One of the reasons I think we are getting large outputs is because of the large variations in scale between the different layers of our network. (Eg. Halite on the map is 0 to 1,000 while presence of a player is 0-1)

For this reason I propose we normalize the inputs of our network.

Double check that our calculation of log_probs is correct and equivalent to PyTorch's distributions model

We should try using the score as the reward and see if optimizing directly for score helps us create better agents.

I'm still worried that the agents will learn to cooperate rather than compete, but it's worth a shot since learning seems to slow with +1 and -1 rewards.

I've littered 12,64,64 all over our code and we should probably just declare these in a single place so it won't be painful if we change the input structure.

500 steps is a lot. Maybe we should use a larger discount rate 0.999?

In order to give the agent experience in different situations, consider randomizing where the dropoff spawns for each player.

Things I would like to do today:

• Finish implementing forward() for the model
• Finish calculating losses
• Run a few training epochs
• (stretch) Print out the losses and scores our agents are attaining

Similar to #1 I need to convert the board into an input for my neural network.

Gradient clipping might help us avoid exploding gradients without having to use such a small learning rate. We should try to implement it.

nn.utils.clip_grad_norm_()

We need an approach for testing out different hyperparameters. Here are some things we should implement/do:

• Print out all hyperparameters at beginning of run
• Print out network information at beginning of run
• Print out mean losses
• Average metrics over longer periods of time (Right now it's 10, try 50?)

• Visualize current progress on one agent
• Set up GCloud
• Run grid search on GCloud
• Visualize input layers
• Plot input layers

Everyone says you should have it, so let's add it and see how it performs.

If we would like to have varied states, we could consider having random offsets for each game. We would have to properly handle wrapping around the edges of the board though.

2000 rollouts * 12 depth * 64 height * 64 width = 98 MB.

We should use a more compact state representation and expand it only when we need it.

We may also wish to investigate using sparse tensors here.

Whenever we get a new "best" network, we should save that network's weights so we can load it from another implementation and watch the replays.

Let's start looking at the results and ruling out certain hyperparameter ranges.

*High score likely due to opponent failing very quickly
** Appears to still be improving

It is probably useful to know how roughly long it takes to train different algorithms. This will likely only be an approximation since the training time will vary based on load but it might be useful to have.

Things we'd like to look at today:

• Create script that plots scores and steps for completed runs when given a log file
• Visualize a few games with hyperparameters that seemed to do well

We're running a pretty costly grid search over hyperparameters right now. It would help to have a script that went through the output files and charted results from each search.

After fixing #6 we should scale down the map to see if that improves anything. At the very least it may make simulations run much faster.

Things I would like to do today:

• Investigate that from_blob is working properly for sampled_log_probs and others
• #8
• Try to guarantee backprop works properly

Now that we're seeing signs of life, we should use a better neural network to control our actions and policy.
I'm thinking to use ResNet-18 from: https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py

We should measure how many self-crashes occur to observe how this number changes when we use different hyperparameters.

• Build a ResNet
• Test ResNet on MNIST
• Count/Plot input layer values
• Optionally change output

If our agent crashes its own ships into one another it might be useful to punish this action with a -0.1 or -0.05 reward on this step/

Our agent needs to be able to restart the game in order to generate new rollouts.

It seems to be something similar to the following:

//NOTE: These seem to be the steps necessary to start a new game!
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~`
hlt::Map map(map_parameters.width, map_parameters.height);
hlt::mapgen::Generator::generate(map, map_parameters);

std::string replay_directory = replay_arg.getValue();
if (replay_directory.back() != SEPARATOR) replay_directory.push_back(SEPARATOR);

hlt::GameStatistics game_statistics;
hlt::Replay replay{game_statistics, map_parameters.num_players, map_parameters.seed, map};
Logging::log("Map seed is " + std::to_string(map_parameters.seed));

hlt::Halite game(map, game_statistics, replay);
//~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

We could add a very small negative reward to turtles who move when they aren't supposed to be able to move. Something like -0.002 might work.

If it has any hope of learning to return Halite to the base, it will do so when there is only one turtle spawned. We should make it only spawn one turtle and see how it performs.

We'll do this after a gridsearch of hyperparameters

Tasks we would like to work on today:

• Finish Batcher class

  • Batcher should take a list of rollouts and allow us to randomly sample batches of them

• Investigate dones

  • Realized last night that we might be representing done incorrectly

• Begin train_network() method