[QUESTION] Splitting big models over multiple GPUs about comet

[vince62s]

idea here. Ctranslate2 just integrated tensor parallelism. It also support XMLRoberta, so just wondering if we could adapt a bit the converter so that we could run the model within CT2 which is very fast.
How different is it from XML Roberta at inference ?

[vince62s]

Does it support XLM-R XL? the architecture also differs from XLM-R

we can adapt if we have the detailed description somewhere.
cc @minhthuc2502

[zwhe99]

same question here

[ricardorei]

Last time I check this was not very easy to do with pytorch-lightning.

We actually used a custom made implementation with FSDP to train these larger models (without using pytorch-lightning). I have to double check if the new versions support FSDP better than the currently used pytorch lightning version (2.2.0.post0).

But short answer: model parallelism is not something we are supporting in the current codebase.

[ricardorei]

Does it support XLM-R XL? the architecture also differs from XLM-R

[ricardorei]

It seems like they improved documentation a lot actually: https://lightning.ai/docs/pytorch/stable/advanced/model_parallel/fsdp.html