Missing non-linearity between ResNet layers? about bringing-old-photos-back-to-life

[zhangmozhe]

In fact, there are several variants of residual block. As discussed in ResNet ECCV paper (https://arxiv.org/pdf/1603.05027.pdf), the activation after the addition (original version) will impede the information propagation and cause performance degradation. You can see detailed comparison of different variants in the paper.

[alechoag]

Thanks for the reply (and the excellent paper and code!). One follow-on question: if the goal is to use a full pre-activation configuration (to preserve the skip connection), shouldn't the norm and activation appear before the convolution in the ResNet?

[zhangmozhe]

Since there are several variants, our implementation does not strictly follow the pre-activation manner, but still remove the activation after the addition operator. Such implementation style is also widely adopted in research work.