bhneo / decorrelated_bn

An implementation of DecorrelatedBN by tensorflow

Decorrelated Batch Normalization

This project provides the Tensorflow implementation of ZCA whitening described in paper:

Decorrelated Batch Normalization(CVPR 2018)

and IterNorm whitening in paper:

Iterative Normalization: Beyond Standardization towards Efficient Whitening(CVPR 2019)

Requirements

• python3
• seaborn
• matplotlib
• easydict
• tensorflow >= 2.0.0

Running experiments

To reproduce the VGG-network experiment, just run vgg.py and pass the config parameters. For example:

python vgg.py --type=A --batch=256 --lr=0.1 --method=zca --m=0

where the "type" denotes the type of VGG-network architecture, "batch" denotes the batch size, "lr" denotes the initial learning rate, "method" denotes the whitening method (zca, iter_norm), "m" denotes the group size (0 indicates full whitening).

Usage

An example can be found in vgg.py.

1. Copy the common/normalization.py to your root directory and import it.
2. Build a DecorelationNormalization layer to replace the batch normalization layer.

from common import normalization

...
feature = normalization.DecorelationNormalization(decomposition='iter_norm_wm',
                                                  iter_num=5)(feature)

Reference

More deteils please refer to the implementations:

• Torch: princeton-vl/DecorrelatedBN
• Pytorch: huangleiBuaa/IterNorm-pytorch

decorrelated_bn's Issues

DecorelationNormalization layer giving nans

Hi,

I've tried to use the DecorelationNormalization layer in a recent version of Tensorflow with my CNN architecture and it gives nan values in the training. I had to make some small changes to the normalization.py file so that it works with my Tensorflow version, so I am not sure whether it is something that I changed or not (you can find my code below). Do you have any insight on that?

https://github.com/fotisdr/decorrelated_bn/blob/master/common/normalization.py

I should add that I tried both 'zca_wm' and 'iter_wm' methods and they both give nans. The former also gives the following error during training: Eigen::BDCSVD failed with error code 3

Thanks a lot!
Fotis

EDIT: If this can be of any help, I've identified the issue to the computed trace getting negative values, which gives nan when computing the tf.sqrt(trace) at the end of the get_inv_sqrt function.