Initial fixes:

the model's predictions were in tensor form. This corrupted the env once the mixing started using the model: the self.cur_pos became an np.array of 1d, 1-element tensor.

Graphs:

as-is, the tensorboard calls plotted the loss on each epoch, and also all the epoch losses on one graph. We'd instead like to have one single big curve for each run, showing how the loss evolves per episode. This will allow us to compare the different curves, and thus optimise the HPs better.

Simply plotting the losses doesn't work, though. It produces incomparable graphs: as time goes on, we collect more and more data samples, meaning the running loss of the last epoch of each episode it strictly bigger, simply because it acts on more observations and actions. This leads to the appearance that the loss is increasing over time. This is an illusion: it is the dataset that increases over time.

We can convince ourselves of this by taking the mean of the losses w.r.t. to the size of the dataset. With that trick, we find the familiar curves normally associated with loss curves.

But that's also kind of bad: the numbers are so small that we're bound to have floating point errors. Instead, I planned on using the number of batches per dataset instead of the size of the dataset in the mean. But that would also be bad, as it would lose meaning if we change the batch size. So we're going to use a magic number instead, just so it doesn't vary with the HPs.

The magic number can be editted.