question about the self-attention in the Encoder about nematus

[rsennrich]

Hi @lovodkin93,

if I understand you correctly, you're asking why self-attention for padded tokens (those beyond end-of-sentence) is not fully disabled, and whether this will affect the model parameters.

The short answer is that the padded states are masked further down in the computation path, and thus have no effect on the loss (there is masking in both the self-attention and the cross-attention that prevents attention to them, and on the decoder side, the cross-entropy loss itself is also masked beyond the end-of-sentence). Consequently, the error signal that is being backpropagated through these connections is 0, and they will not affect the model parameters.

[lovodkin93]

great, I figured as much, but didn't find the relevant further masking, and wanted to make sure and not just assume.
Thank you!