universome / loss-patterns Goto Github PK

Very inspiring work!

I was wondering if you thought about checking for arbitrary patterns close to some trained weights (i.e. constrain the weights to stay close to some constant weights, for example with an L2 loss). The trained weights could be snapshots from standard training (e.g. SGD, Adam), either after training or in the middle of it.

The reason is what one could ask how "far from the beaten path" these patterns are; if they're ubiquitous and will be commonly found during training, or if you need to move in weight-space very far to regions that are not very likely to occur with current training practices.