Increasing WER & Validation Loss During Whisper Fine-Tuning about community-events

[sanchit-gandhi]

Hey @monk1337! Awesome that you reduce the WER by over half in just 1k training steps! The increasing WER after 1k steps looks like it could well be a case of over-fitting. You could combat this by:

1. Introducing regularisation through dropout and activation dropout (set these config attributes to either 0.1 or 0.2 to activate dropout. In my experience, a small amount of dropout helps for small datasets. Going above a dropout of 0.2 is too severe and hurts performance)
2. Using a larger dataset (in practice this may not be feasible, but it is one valid solution to reduce over fitting)

If you have the script you used to train the model and pushed the checkpoint to the Hugging Face Hub, I'd be happy to advise further!