Asking about the default config option `save_interval: 5` when pretraining on AlphaDB about gearnet

[Oxer11]

Hi, this is a good question. Here we use Round-Robin sampling for pre-training, since it's impossible to load the entire pre-training dataset into the memory. Ideally, we should set save_interval: 1 so that we can iterate over all pickles per epoch. But it takes time to load each pickle into the memory. So we use the save_interval to balance the time of data loading and the training performance. With a small save_interval, we can simulate the ideal round-robin sampling; with a large save_interval, we can reduce the time to load datasets.

Note that if you simply train enough number of epochs on one pickle and then move to the next one, you may face the catastrophic forgetting problem, i.e., forgetting last pickle after training on new pickles.

[yunxiaoliCB]

Thank you for the explanation! That makes perfect sense.