glot500-large about glot500

[kargaranamir]

Are there plans to train a large glot500 model? Thanks for your work so far!

Hi, thank you for opening this issue. We made some attempts at training, but unfortunately, due to unresolved issues and being GPU poor, the training was not successful. Regrettably, the situation will not change in the foreseeable future.

[Bachstelze]

PEFT could be a solution to lower the hardware consumption and it would be more adaptable. At the moment MLM isn't tested with it:
huggingface/peft#116
Yet there isn't a reason why it shouldn't work.

[kargaranamir]

PEFT could be a solution to lower the hardware consumption and it would be more adaptable. At the moment MLM isn't tested with it: huggingface/peft#116 Yet there isn't a reason why it shouldn't work.

cc @lpq29743 @ayyoobimani