RuntimeError: Error(s) in loading state_dict for FastTransformer: about dl4mt-nonauto

[mohengzxr]

Decoding from NA Models why? --load_vocab,Why does it appear size mismatch for encoder.out.weight: copying a param with shape torch.Size([36377, 278]) from checkpoint, the shape in current model is torch.Size([38022, 278]).???

[mohengzxr]

@mansimov

[mansimov]

Hm it is a bit hard for me to figure out why this happens.
Haven't used this codebase for about a year or so...

[mohengzxr]

--lr_schedule anneal and transformer different?
@

[mohengzxr]

@mansimov

[mansimov]

--lr_schedule anneal just anneals the learning rate starting from certain value
--lr_schedule transformer first warmups up the learning rate and then anneals it down. by warming up I mean it increases the learning rate from very low value to certain value like 3e-4 for first couple of training iterations. see original transformer paper for reference

[mohengzxr]

anneal and transformer ?Whether it affects Bleu?
@mansimov

[mohengzxr]

Why is the Bleu produced by choosing ananne and transformer different?
What is their impact on bleu?
Thank you
@mansimov

[mansimov]

Anneal works for well for IWSLT, transformer works well for WMT.
In general I advice to use transformer schedule.
Don't remember how exact impact of choosing learning rate schedules is on bleu

[mohengzxr]

In the experiment, we chose the Uyghur-Chinese corpus.
When using ananne and transformer, the Bleu of the two is very different.
I don't understand very much.

[mohengzxr]

I am very interested in this paper, I want to have a little research on your basis.

[mohengzxr]

I really hope to get your help.

[mansimov]

I am glad that you are interested :)

It is very hard to say why results are different without diving into details. But it is unsurprising to me that with Transformers you need to be careful with learning rate schedule depending on number of GPUs you have, model size and dataset size!

Good luck!

[mohengzxr]

@jasonleeinf @mansimov @kyunghyuncho
Hi
In your paper,
Equation 2: t=1,l=0,l-1=-1 is wrong;
Excuse me, how did you explain it? I hope to help you.

[mohengzxr]

[mansimov]

Hi @mohengzxr

Thanks for pointing out this mistake in the paper. l=1 instead of l=0 in the first sum.
Best, Elman