[BUG] Why does increasing model_max_length result in fine-tuning not working? about autotrain-advanced

[jackshiwl]

@abhishekkrthakur able to give insight on this? seems like a major bug..

does it mean increasing the model_max_length (or block_size) while keeping the data length the same will affect the fine tuning process?

[jackshiwl]

@abhishekkrthakur sorry, any insights on this?

It seems like when I increase the block-size / model_max_length during fine-tuning to be much greater than the input token length, the model is not able to learn anymore from the fine-tuning (even though its severely overfitted).

[abhishekkrthakur]

please be patient @jackshiwl . many times, immediate response is not possible :)
if your sentences are small and you are using large max len, it means there will be too many padding tokens, which may account for the model not learning properly. given your data, you should choose the best hyperparameters suitable for the model you are training. this is not a bug.

[jackshiwl]

@abhishekkrthakur,

1. for the padding args, I have it set as default, so it is 'none'. but it doesnt work if padding=right / padding = left.
2. Even if there are paddings, I have tried to overfit it severely, by setting lots of epochs, etc. The loss goes down to abysmally small value, but it is still not able to recall the sample dataset (there is only 1 sample x42 times)
3. i have begin testing from 1024, 2048, ... it all works. But once it hits 4096, it just totally stops recalling, even if I tried to increase the epochs drastically etc.

I am just worried if this is an issue if I use padding=none (default) in my finetuning process? because i have samples that are about 500 tokens, but some are also 4096 (all are trimmed to 4096 max). not sure if this will be a problem for fine-tuning. do you use padding for your own finetuning?

[jackshiwl]

would appreciate if you can elaborate a little on what padding sides do you use for your own finetuning - and also for inferencing.

[github-actions]

This issue is stale because it has been open for 15 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 2 days since being marked as stale.