Question about why the add&norm structure of the tranformer network differ from the typical transformer one about centerformer

[edwardzhou130]

I used prenorm inside each layer.

centerformer/det3d/models/utils/transformer.py

Lines 218 to 238 in 96aa375

	PreNorm(
	dim,
	Attention(
	dim,
	heads=heads,
	dim_head=dim_head,
	dropout=dropout,
	out_attention=self.out_attention,
	),
	),
	PreNorm_CA(
	dim,
	Cross_attention(
	dim,
	heads=heads,
	dim_head=dim_head,
	dropout=dropout,
	out_attention=self.out_attention,
	),
	),
	PreNorm(dim, FeedForward(dim, mlp_dim, dropout=dropout)),

[Liaoqing-up]

I used prenorm inside each layer.

centerformer/det3d/models/utils/transformer.py

Lines 218 to 238 in 96aa375

	PreNorm(
	dim,
	Attention(
	dim,
	heads=heads,
	dim_head=dim_head,
	dropout=dropout,
	out_attention=self.out_attention,
	),
	),
	PreNorm_CA(
	dim,
	Cross_attention(
	dim,
	heads=heads,
	dim_head=dim_head,
	dropout=dropout,
	out_attention=self.out_attention,
	),
	),
	PreNorm(dim, FeedForward(dim, mlp_dim, dropout=dropout)),

I see, but I wonder if you have tried Add&Norm after each layer, which means the residual skip connect input are the features already passed through the Norm. Is it possible that the results of these two structures do not differ much?

[edwardzhou130]

Sorry, I haven't tried Add&Norm after each layer. Do you have experience with this before and would the results be better if you used this implementation?