yukkyo / pytorch-filterresponsenormalizationlayer

PyTorch implementation of "Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks"

pytorch-filterresponsenormalizationlayer's Issues

RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED

Not sure why this occurs.

Setting torch.backends.cudnn.enabled=False fixes this issue. Unfortunately it also increases memory consumption.

bnrelu_to_frn is not work now !!!

bnrelu_to_frn is not work if definition order is different from forward order.

tau and epsilon

Hello, thanks for your work.
I have read your codes and found something mismatching with original paper.
The first is the 'tau' in TLU. In paper, it's said that tau is a vector of size num_channels. But in your codes, It's just a single value.
The second is the epsilon. In paper, it's said when using in conv, epsilon is a constant value of 10^-6. but in your codes , it's a learnable parameter.
Will these mismatches hurt the preformance?

senet_frn first stage bn or frn?

https://github.com/yukkyo/PyTorch-FilterResponseNormalizationLayer/blob/master/senet_frn.py#L287

        if input_3x3:
            layer0_modules = [
                ('conv1', nn.Conv2d(3, 64, 3, stride=2, padding=1,
                                    bias=False)),
                ('bn1', nn.BatchNorm2d(64)),
                ('relu1', nn.ReLU(inplace=True)),
                ('conv2', nn.Conv2d(64, 64, 3, stride=1, padding=1,
                                    bias=False)),
                ('bn2', nn.BatchNorm2d(64)),
                ('relu2', nn.ReLU(inplace=True)),
                ('conv3', nn.Conv2d(64, inplanes, 3, stride=1, padding=1,
                                    bias=False)),
                ('bn3', nn.BatchNorm2d(inplanes)),
                ('relu3', nn.ReLU(inplace=True)),
            ]
        else:
            layer0_modules = [
                ('conv1', nn.Conv2d(3, inplanes, kernel_size=7, stride=2,
                                    padding=3, bias=False)),
                ('bn1', nn.BatchNorm2d(inplanes)),
                ('relu1', nn.ReLU(inplace=True)),
            ]
        # To preserve compatibility with Caffe weights `ceil_mode=True`
        # is used instead of `padding=1`.
        layer0_modules.append(('pool', nn.MaxPool2d(3, stride=2,
                                                    ceil_mode=True)))

Sorry but I'm a little confused. When using FRN in senet, should all batch norms be replaced by FRN?