The fine tuning script run_speech_recognition_seq2seq_streaming.py use interleave_datasets which will truncate the train split about community-events

[sanchit-gandhi]

We need to use interleave_datasets for streaming datasets. Here we do not know the length of each dataset a-priori, and so mix them on-the-fly based on the sampling probabilities that we define, potentially truncating individual datasets when we completely iterate over one of datasets (see "stopping strategies" in the docs).

Whereas we use concatenate_datasets for non-streaming datasets, since we know the lengths of each dataset a-priori, so can mix them entirely). See docs.

[sanchit-gandhi]

Ideally, this is the kind of logic that we want to implement, borrowed from the Distil-Whisper training code: https://github.com/huggingface/distil-whisper/blob/914dcdf3919552d5a3826a9d5db99b059ddcc16e/training/run_distillation.py#L600