Different WERs when decoding with different batch size (--max-sentences) about espresso

[freewym]

Yes, I also observed such things before. Yes I also think it's caused by the beam search code sequence_generator.py. I haven't investigate into this as the code is almost from fairseq, Not sure if people doing MT with fairseq have encountered the same problem.

[joyolee]

Yes, I also observed such things before. Yes I also think it's caused by the beam search code sequence_generator.py. I haven't investigate into this as the code is almost from fairseq, Not sure if people doing MT with fairseq have encountered the same problem.

Thanks for your prompt reply. I might have a check if the people in MT also have similar things later, or submit an issue in fairseq.

[freewym]

The thread in #55 may suggest the reason: the encoder's convolution may give different output when input features are padded differently in different batches

[joyolee]

I would ask if this problem is fixed? I submitted an issue in fairseq, and the author suspects that this problem could be caused by the missing padding mask after each layer. https://github.com/pytorch/fairseq/issues/3078#issuecomment-754858923

[freewym]

After looking at the issue I mentioned in my last post, I agree that the encoder's padding is the cause. But I think it only happens with convolutions (not confirmed yet): the shorter examples in a batch will be padded with different padding frames in different batches, making the convolution kernel being applied across the example boundary get different numbers.