FLOPs about the attention scale about vit-anti-oversmoothing

[peihaowang]

Hi Jiahao,

Thanks for this good question. This piece of code is mostly adopted from SwinTransformer with one line added for AttnScale. The AttnScale can be regarded as a linear combination between the original attention map and a fixed matrix, so its complexity should be $O(H N^2)$. Hope you find this useful.

Peihao

[techmonsterwang]

Thanks for the reply.

It seems like flops += self.num_heads * N * N correspronds to Eq.9 in the paper, and the computation of Eq.7 (mean operation) is ignored?

[peihaowang]

Hi Jiahao,

I have re-examined the code. I guess what you suggested here is correct. The AttnScale has two steps:

1. Decomposes a high-pass filter, which takes num_heads * N * N
2. Rescale and merge two components. This takes 2 * num_heads * N * N.

This enlarges the flops in that step by a constant. I will update the code and numbers in the paper later.

Peihao