Trying to pretrain mBART model. about yanmtt

[prajdabre]

Hi,
Your command is bound to give errors. You are missing a few things:

--langs hi_IN (since there's no language token for Sanskrit you may have to use the one for Hindi. Since you don't provide a language token the code bugs out as it uses some default language token that mbart tokenizer doesn't recognize and segments it weirdly.)

Don't use fp16 if you use multiple GPUs. Regardless I've had mixed results with fp16 so I'd avoid it.

--batch_size 16 (I think you meant this to be 16 sentences. So you need the flag --batch_size_indicates_lines. By default this means number of tokens in batch.)

Additionally, be careful about learning rates and dropouts etc. Good luck!

[nikhilbyte]

Hi,
Thanks for your reply.
I added the arguments.
!python pretrain_nmt.py -n 1 -nr 0 -g 1 --use_official_pretrained --langs hi_IN --batch_size_indicates_lines --pretrained_model "facebook/mbart-large-50" --model_path "facebook/mbart-large-50" --tokenizer_name_or_path "facebook/mbart-large-50" --mono_src "/content/yanmtt/cleaned_Sanskrit_text_for_LM.txt" --shard_files --batch_size 1
You see,I've put the batch size as 1 and even them I'm getting this:

RuntimeError: CUDA out of memory. Tried to allocate 978.00 MiB (GPU 0; 15.90 GiB total capacity; 14.69 GiB already allocated; 337.75 MiB free; 14.76 GiB reserved in total by PyTorch)

GPU:

[prajdabre]

Hi,

I can't really help with the GPU memory issue. 16 GBs is a tab bit small. The only thing you can try is limit the maximum sequence length via the --hard_truncate_length option. Find out what's the average sequence length in your corpus and then play with that argument. Btw why not try IndicBart which is much about a third of mbarts size and is more suited for Indic languages? Since its compact you will not run into memory issues.

[nikhilbyte]

Sure, Thanks.