about the Conv2DLayer in DenseNet about recipes

[benanne]

The nonlinearities are added separately, try to search for NonlinearityLayer in the code.

Presumably this is because it is easier to reason about densenets if you consider a block to consist of (batchnorm -> nonlinearity -> linear layer), rather than the commonly used definition of (linear layer -> batchnorm -> nonlinearity), which is also the one that Lasagne assumes.

[rongrongxiangxin]

@benanne Thanks, I got what you mean. Just like in
, the structure in the block is (batchnorm -> nonlinearity -> linear layer).

But, in the initial convolution,

and in the linear layer in the block, they are linear layers. So the linear layer is special in densenet? Because in other networks, the convolution layers are always nonlinear.

[f0k]

Sorry, I was working on a paper. The initial convolution has nonlinearity=None because it is followed by bn_relu_conv(), which adds batch normalization and a rectifier. If the initial convolution had a rectifier itself, the network would start with conv -> relu -> bn -> relu, which is not what we want. Feel free to print the layers after creating the network:

for layer in lasagne.layers.get_all_layers(network):
    print layer.name, layer.__class__.__name__

It may get confusing due to the merge layers, but at least the beginning should be clear.

Presumably this is because it is easier to reason about densenets if you consider a block to consist of (batchnorm -> nonlinearity -> linear layer)

Exactly, like in the pre-activation residual networks.