sdoria / simpleselfattention Goto Github PK

Hi Sebastian, thanks for the layer.

Can I clarify what you mean by:
"the network converges if SimpleSelfAttention is placed right after a convolution layer that uses batch norm"

Are you saying the order of operations should always be

Conv => Batchnorm => Self Attention ?

And activation should come afterwards?

@sdoria
Hi ,thank you for your work, is great.
I have a question ,
SimpleSelfAttention and se (senet ) , se is a independent structure,
Which is more effective? Have you done any experiments.

self.gamma = nn.Parameter(tensor([0.]))

o = self.gamma * o + x
or
o = self.gamma * torch.bmm(h, beta) + x

Hi @sdoria ,

I have a problem understanding the code. If we set gamma to 0., then wouldn't that always be zero ?

where is the atention ? As gamma is zero the complete self-attention is zero and the code finally adds input and pass it to linear.

I dont see the self attention here.

May be i did not understand it clearly, would you mind clarifying that?

Thanks!