About Layernorms within MSAB about mst

[caiyuanhao1998]

we aim to stabilize the spectral-wise self-attention map produced by Q and K. For V, we apply mask-attention and use LN to process it in the subsequent layers. I can also try to apply an LN to process V in the MS-MSA module.

[caiyuanhao1998]

Hi,

Thanks for your interests. Another layer normalization is in Line 150-151 of MS_MSA module in MST.py, check this

[yyyang-2019]

Thanks for your quick reply and i hope that won't bother you.
So the normalize is only applied on Q and K in computing attention matrix, leaving V not normed. That's different from the LN in common transformer-based models which applies to the whole feature map. I wonder what the reason this design is for?

[yyyang-2019]

okay i got it. appreciate your great work :)