How to achieve exact same # of samples seen? about datacomp

[zwsjink]

Cool! Then your understanding here seems correct!

OK, thanks for your reply, so in my example, I would set --train-num-samples to 12.8M while set --epochs to 10. Alternatively, I can also do it with 25.6M train-num-samples and 5 epochs , right? as long as the multiplication meets the same, there should be no difference in the final training performance , I suppose ?

Great, thanks for your time. I will let you know once we reproduce your conclusion . Besides, good luck to your PhD degree pursuit. :D

[sagadre]

@mingtan2 yes! that should be fine!

[sagadre]

Hi @zwsjink, the number of epochs controls the number of checkpoints that are saved during training. If training on k samples (e.g., k = 128M for the medium pool) with number of epoch n, we will save a checkpoint after k // n samples are seen. Hence each epoch corresponds to seeing k // n samples from the training pool with replacement.

See here where number of samples per epoch are set.

See here where number of epochs is set to be the number of checkpoints.

[sagadre]

In your example of the 30M dataset for the medium scale with number of epochs = 10. Each epoch would correspond to sampling 12.8M samples from the 30M dataset (with replacement)

[zwsjink]

In your example of the 30M dataset for the medium scale with number of epochs = 10. Each epoch would correspond to sampling 12.8M samples from the 30M dataset (with replacement)

OK, thanks for your reply, so in my example, I would set --train-num-samples to 12.8M while set --epochs to 10. Alternatively, I can also do it with 25.6M train-num-samples and 5 epochs , right? as long as the multiplication meets the same, there should be no difference in the final training performance , I suppose ?

[sagadre]

For participating in DataComp, you don't have to set --train-num-samples or --epochs directly. Please see this section of the README for a sample command line, where $scale would be medium for the 128M pool.

You can additionally set the --num_checkpoints flag as seen here to specify how many checkpoints you would like to save. Our code will take care of setting --train-num-samples and --epochs accordingly under the hood.

Hope this helps!

[sagadre]

As for the performance deltas for setting different --num_checkpoints, there should not be dramatic changes in downstream performance. At the start of every "epoch" the dataloader is re-initialized, hence different values for --num_checkpoints will lead to different data orders similar to changing the random seed. Please see here for a note on seed variance.

[zwsjink]

For participating in DataComp, you don't have to set --train-num-samples or --epochs directly. Please see this section of the README for a sample command line, where $scale would be medium for the 128M pool.

You can additionally set the --num_checkpoints flag as seen here to specify how many checkpoints you would like to save. Our code will take care of setting --train-num-samples and --epochs accordingly under the hood.

Hope this helps!

Well currently i'm not planning to participate in the track, just trying to follow the paper and do something very similar with you guys' OPEN_CLIP & CLIP_BENCHMARK toolbox on different dataset.

[sagadre]

Cool! Then your understanding here seems correct!

OK, thanks for your reply, so in my example, I would set --train-num-samples to 12.8M while set --epochs to 10. Alternatively, I can also do it with 25.6M train-num-samples and 5 epochs , right? as long as the multiplication meets the same, there should be no difference in the final training performance , I suppose ?

[mingtan2]

As for the performance deltas for setting different --num_checkpoints, there should not be dramatic changes in downstream performance. At the start of every "epoch" the dataloader is re-initialized, hence different values for --num_checkpoints will lead to different data orders similar to changing the random seed. Please see here for a note on seed variance.

@sagadre Had this same question. Thanks for explaining here. Then, it is allowed to set the num_checkpoints to be 1 (instead of 8) to accelerate training without reinitializing dataloader?

[mingtan2]

@mingtan2 yes! that should be fine!

@sagadre In addition, is it allowed to disable dataset_resampled here if it is compatible with DataComp challenge settings?

[sagadre]

Hi @mingtan2, yes you should keep the --dataset_resampled for the challenge