brycexu / mr-residual-net

Evaluations of Merge and Run in Binarized Residual Neural Networks

This repository contains my evaluations of Merge and Run in Binarized Residual Neural Work

Copyright: Xianda Xu [email protected]

Evaluation on Cifar-10

Accuracy	Full-Precise (1)	Binarized (1)	Binarized (2)	Netscope
ResNet-18	93.28%	90.50%		Network
MR-ResNet-20	92.15%	87.77%	90.48%	Network
MR-ResNet-32	93.39%	90.46%	92.58%	Network

Style 1 does downsampling by using concatenation and convolusion (kernel_size=1, stride=2, padding=0)

Style 2 does downsampling by using concatenation and average pooling (kernel_size=2, stride=2)

Binarization Principle

• Keep full-precision on the first convolutional layer and the last linear layer.

• In binarized convolutional layers, all weights are binarized and scaled in propagation (https://arxiv.org/abs/1603.05279). But here, the scale factors are not learnt but all set to 1.

• BatchNorms in binarized blocks have no affine weights and bias parameters.

• Since activations are not binarized, ReLU is used instead of HardTanh (https://arxiv.org/pdf/1602.02830).

Baseline

Model: ResNet-18

Paper: (https://arxiv.org/abs/1512.03385)

Netscope: Network

Full-Precise Accuracy on Cifar-10: 93.28% with 80 epoches

Binarized Accuracy on Cifar-10: 90.50% with 80 epoches

Merge and Run

Model: MR-ResNet-20 (the number of layers is almost identical to the baseline ResNet-18 model)

Paper: (https://arxiv.org/abs/1611.07718)

Netscope: Network

Full-Precise Accuracy on Cifar-10: 92.15% with 80 epoches

Binarized Accuracy on Cifar-10: 87.77% with 80 epoches

• Downsampling is done by firstly concatenating left-branch and right-branch and secondly using a convolusion (kernel-size:1, stride:2, padding:0)

Binarized Accuracy on Cifar-10: 90.48% with 80 epoches

• Downsampling is done by firstly concatenating left-branch and right-branch and secondly using a average pooling (kernel_size:2, stride:2)

Model: MR-ResNet-32 (the depth is identical to the baseline ResNet-18 model)

Paper: (https://arxiv.org/abs/1611.07718)

Netscope: Network

Full-Precise Accuracy on Cifar-10: 93.39% with 80 epoches

Binarized Accuracy on Cifar-10: 90.46% with 80 epoches

• Downsampling is done by firstly concatenating left-branch and right-branch and secondly using a convolusion (kernel-size:1, stride:2, padding:0)

Binarized Accuracy on Cifar-10: 92.58% with 80 epoches

• Downsampling is done by firstly concatenating left-branch and right-branch and secondly using a average pooling (kernel_size:2, stride:2)

mr-residual-net's Issues

The question about the number of layers in the binarized model

Hello, I wonder know why the binarized ResNet has fewer layers compared with full precision ResNet. Shouldn't we keep the number of their layers the same?