General information about distillation about dl4mt-nonauto

[jaseleephd]

Hi Lucas! We're actually using sequence-level distillation, so we take the output from a well-trained autoregressive model and use that as the ground truth target.

Have a look at https://github.com/nyu-dl/dl4mt-nonauto/blob/master/run.py#L327-L328, we saved the AR output in a separate file, and load from that file, instead of the actual ground truth.

[jaseleephd]

Yeah that's right, we used output from the autoregressive transformer (with beam search).

yeah, we're not the only ones :)
https://arxiv.org/pdf/1711.10433.pdf
https://arxiv.org/pdf/1711.02281.pdf
https://arxiv.org/pdf/1902.03249.pdf

[pclucas14]

Wow big thanks for the share! I've been working for the past two months on my version of insertion transformer (minus the distillation). Glad to be aware of this now rather than in two months :p

[mansimov]

I actually played with self distillation during summer on WMT'14, that is use the output of non-autoregressive model at iteration 20 as groundtruth after non-autoregressive model more-or-less converged (exactly what @jasonleeinf has described).

I found that it improved BLEU score significantly on first 1-3 iterations of non-autoregressive decoding, but in the latter iterations (like iteration=20) the results only improved by very small score about 0.5 BLEU

[pclucas14]

Thanks for the link! So the target is still a one-hot distribution (resulting from beam search on the teacher instead of the actual target) ?
The fact that you guys get a significant increase in performance from this is quite cool!

[pclucas14]

What are your thoughts on self distillation ? Did you find that it helped the model converge to a better solution ?

[jaseleephd]

do you mean taking the output from the non-autoregressive model and feeding back to itself as a training example? we haven't tried that, but i doubt it'll work better than using the output from an AR model

[pclucas14]

Cool thanks! I was curious because it was in the code, but not in paper :p

[kyunghyuncho]

self-distillation exploits some weird flaw of our approximate decoding. if we used e.g. sampling from the model distribution, on expectation it really shouldn't do much: see https://twitter.com/DeepSpiker/status/1096539984738377734

[pclucas14]

Interesting discussion! However does this apply here also ? I would assume that the distribution after 1 vs 20 iterations is different, therefore (even in expectation) self-distillation could provide useful gradients

[mansimov]

Plus the weights of the model at iteration 1 and iterations > 1 are not shared