Layernorm in Cross attention about vit-pytorch

[turtleman99]

Gotcha. Thank you so much for your explanations! :)

[lucidrains]

it has been a while, but can you check to see if the context being received isn't coming from another transformer with a final layernorm?

[turtleman99]

oh, this is a good point. My x and context are actually features from pre-trained ViT models. I believe I can remove layernorm for x and context, right?

[lucidrains]

@turtleman99 if you are following the same scheme as cross vit, then the code is correct and stays the same. x is layernormed (pre-layernorm configuration + residual), while context is left alone since it is already layernormed from your pretrained ViT model. You would only need to layernorm the context if the context also cross attends to x and is updated, like in the ISAB architecture, but that isn't what the cross vit authors did