The question of the gated axial attention formula about medical-transformer

[jeya-maria-jose]

Hi @charleneolive , the gate parameters are not updated for the initial epochs to give an early start to the other parameters. From experiments, we found this configuration to work better. Later the requires_grad is set as True from 10th epoch.

Medical-Transformer/train.py

Line 169 in 10fb5a7

if epoch == 10:

[charleneolive]

Hello @jeya-maria-jose and @wang-yu-xuan , from what I understood from the code, the gated parameters Gq, Gk, Gv1 and Gv2 is just self.f_qr, self.f_kr, self.f_sve and self.f_sv on the code is it? They appear to be nn.Parameters with value of 0.1 and requires_grad = False. Does it mean that these parameters are not updated during training? Why then does Section 3.2 call them learnable parameters? How did you decide on the value of 0.1?

[charleneolive]

Hi @charleneolive , the gate parameters are not updated for the initial epochs to give an early start to the other parameters. From experiments, we found this configuration to work better. Later the requires_grad is set as True from 10th epoch.

Medical-Transformer/train.py

Line 169 in 10fb5a7

if epoch == 10:

Makes sense, thank you!