Ideal number of epochs? Number of examples meaning? about finetune-gpt2xl

[Xirider]

Hi,
you can change how the loss is evaluated against your eval set with --eval_steps. In the example it is evaluated every 15 steps. As long as the eval loss goes down, usually the model will improve. Sadly there are no good rules for how many epochs you need. For me, everything between 1 - 15 epochs worked well, depending on how much data i have and how much i want to overfit. Just set --save_steps to 500 and test each checkpoint for yourself.

The reasons why the number of examples is lower than your training and eval texts, is that run_cml.py concatenates all your texts with EOS (End of Sequence) tokens in between. Then the long string gets split into equal parts of your defined block_size (check line 374 of run_clm.py). You are not loosing any data.