jack-willturner / deep-compression

Learning both Weights and Connections for Efficient Neural Networks https://arxiv.org/abs/1506.02626

Learning both Weights and Connections for Efficient Neural Networks

A PyTorch implementation of this paper.

To run, try:

python train.py --model='resnet34' --checkpoint='resnet34'
python prune.py --model='resnet34' --checkpoint='resnet34'

Usage

The core principle behind the training/pruning/finetuning algorithms is as follows:

from models import get_model
from pruners import get_pruner 

model = get_model("resnet18")
pruner = get_pruner("L1Pruner", "unstructured")

for prune_rate in [10, 40, 60, 80]:
    pruner.prune(model, prune_rate)

We can choose between structured/unstructured pruning, as well as the pruning methods which are in pruners (at the time of writing we have support only for magnitude-based pruning and Fisher pruning).

Bring your own models

In order to add a new model family to the repository you basically just need to do two things:

1. Swap out the convolutional layers to use the ConvBNReLU class
2. Define a get_prunable_layers method which returns all the instances of ConvBNReLU which you want to be prunable

Summary

Given a family of ResNets, we can construct a Pareto frontier of the tradeoff between accuracy and number of parameters:

Han et al. posit that we can beat this Pareto frontier by leaving network structures fixed, but removing individual parameters:

deep-compression's Issues

Hi, I can't import tinynet

Could you updata the tinynet's file and the URL of tinynet.pth.tar? Thanks for your excellent work.

I have a question about parameter number.

after pruning, I count the parameter of the network resnet34.
well... I found the number of parameters to be 4.2 * 10e7
I know parameter number of resnet34 is about 2 * 10e7
I think it's because of the mask of layer that the difference is about double, right?
then how to remove the mask layer after prunning?
Thanks you.

Transposed Convolution

can I use your proposed algorithm for pruning weights in Transposed Convolution layers, e.g. in DCGAN?

Hi,I can

A question for channel pruning on mobilenet

Thank you,your code enlightens me.I have trained mobilenet on CIFAR10,and then I got 87% acurracy rate.Then I use deep_compress function,after i use sparsify function,I have found the weights of kernel in some layer are all zero,it means when I use channel prue,I will loss all the following layer? Have i done something wrong?Thank you for your time!

Error in structured pruning

Hi, when I prune resnet in structured type, I got error like this:

Traceback (most recent call last):
File "prune.py", line 131, in
pruner.prune(model, prune_rate)
File "/content/deep-compression-master/pruners/l1_pruner.py", line 70, in prune
self.structured_prune(model, prune_rate)
File "/content/deep-compression-master/pruners/l1_pruner.py", line 30, in structured_prune
threshold = np.percentile(channel_norms, prune_rate)
File "<array_function internals>", line 6, in percentile
File "/usr/local/lib/python3.7/dist-packages/numpy/lib/function_base.py", line 3868, in percentile
a, q, axis, out, overwrite_input, interpolation, keepdims)
File "/usr/local/lib/python3.7/dist-packages/numpy/lib/function_base.py", line 3988, in _quantile_unchecked
interpolation=interpolation)
File "/usr/local/lib/python3.7/dist-packages/numpy/lib/function_base.py", line 3564, in _ureduce
r = func(a, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/numpy/lib/function_base.py", line 4103, in _quantile_ureduce_func
)), axis=0)
ValueError: operands could not be broadcast together with shapes (128,) (64,)

It seems because some element's lenth in channel_norms list are different. What should I do?