matthewalmeida / pacs_sampling_experiments Goto Github PK

I'm seeing some models (esp. those with relatively high learning rate) have a "best" model from too early in training; these seem to overfit the validation set to some degree. Given the problem domain, there is little advantage to fitting the validation set hard.

Add dropout with flag before the top linear projection layer to pull back on the training fit a little bit. We want to try longer, slower training with higher regularization to see if we can get some more stable results against the validation set.

These seem to coincide in all cases, but there would be a potential issue if for some reason they didn't - better to use cumulative_sizes everywhere.