wangxu930123 / sparse_switchnorm

Sparse Switchable Normalization with sparse activation function SparestMax

Sparse Switchable Normalization In Image Classification

This is the PyTorch implementation of the paper SSN: Learning Sparse Switchable Normalization via SparsestMax, CVPR 2019.

By Wenqi Shao, Tianjian Meng, Jingyu Li, Ruimao Zhang, Yudian Li, Xiaogang Wang, Ping Luo.

Paper-arxiv

Sparse Switchable Normalization

Sparse Switchable Normalization is able to learn only one normalization operation for each normalization layer in a deep neural network in an end-to-end manner.

Comparisons of top-1 accuracies on the validation set of ImageNet, by using ResNet50 trained with SSN, SN, BN, and GN in different batch size settings. The bracket (·, ·) denotes (#GPUs,#samples per GPU).

Normalizer	(8,32)	(8,16)	(8,8)	(8,4)	(8,2)
BN	76.4	76.3	75.5	72.7	65.3
GN	75.9	75.8	76.0	75.8	75.9
SN	76.9	76.7	76.7	75.9	75.6
SSN	77.2	77.0	76.8	76.1	75.9

Getting Started

• Install PyTorch
• Clone the repo:

  git clone https://github.com/switchablenorms/Sparse_SwitchNorm.git
  

Requirements

• python packages
  • pytorch>=0.4.0
  • torchvision>=0.2.1
  • tensorboardX
  • pyyaml

Data Preparation

• Download the ImageNet dataset and put them into the {repo_root}/data/imagenet.

Training a model from scratch

./train.sh

Number of GPUs and configuration file to use can be modified in train.sh

Evaluating performance of a model

Download the pretrained models from Model Zoo and put them into the {repo_root}/model_zoo

./test.sh

Or you can specify the checkpoint path by modifying test.sh

--checkpoint_path model_zoo/ssn_8x2_75.848.pth \

Model Zoo

We provide models pretrained with SSN on ImageNet. The configuration of SSN is denoted as (#GPUs, #images per GPU).

Model	Top-1*	Top-5*	Download	MD5
ResNet50v1+SSN (8,32)	77.25%	93.29%	[Google Drive] [Baidu Pan (pin:ckd2)]	5c9fb111577b040e62461db51ffce69b
ResNet50v1+SSN (8,16)	76.98%	93.29%	[Google Drive] [Baidu Pan (pin:995u)]	d7085632529a9b1945a28a4d3bf4cacb
ResNet50v1+SSN (8,8)	76.75%	93.33%	[Google Drive] [Baidu Pan (pin:k9ay)]	af85a39920644421fc48e216aba6ff0e
ResNet50v1+SSN (8,4)	76.07%	96.71%	[Google Drive] [Baidu Pan (pin:jpvu)]	949d4bd54a7c63a3371b941eb4d3ea69
ResNet50v1+SSN (8,2)	75.85%	92.73%	[Google Drive] [Baidu Pan (pin:pdyx)]	dadb0f1a0c49c31aedd6cb83d4996a03

*single-crop validation accuracy on ImageNet (a 224x224 center crop from resized image with shorter side=256)

In evaluation, download the above models and put them into the {repo_root}/model_zoo.

Citation

If you find this work helpful in your project or use our model zoo, please consider citing:

@article{shao2019ssn,
  title={SSN: Learning Sparse Switchable Normalization via SparsestMax},
  author={Shao, Wenqi and Meng, Tianjian and Li, Jingyu and Zhang, Ruimao and Li, Yudian and Wang, Xiaogang and Luo, Ping},
  journal={arXiv preprint arXiv:1903.03793},
  year={2019}
}