incheon-cho / dynamic_model_pruning_with_feedback

Implement of Dynamic Model Pruning with Feedback with pytorch

Dynamic Model Pruning with Feedback

Paper Link : Dynamic Model Pruning with Feedback - ICLR2020

It's UNOFFICIAL code!

If you want to get information of hyperparameters, you should read appendix part of this paper

Abstract

(1) Allowing dynamic allocation of the sparsity pattern

(2) Incorporating feedback signal to reactivate prematurely pruned weights

Method

Run

python main.py cifar10 --datapath DATAPATH --a resnet layers 56 -C -g 0 save train.pth \
--epochs 300 --batch-size 128  --lr 0.2 --wd 1e-4 --nesterov --scheduler multistep --milestones 150 225 --gamma 0.1

Experiment

	Best Top-1 Acc(%)	Sparsity(%)
Basline	93.97	0
DPF	93.73	90.00

Experiment on ResNet56 for CIFAR10

DPF run :

python main.py cifar10 --datapath DATAPATH -a resnet --layers 56 -C -g 0 --save prune.pth \
-P --prune-type unstructured --prune-freq 16 --prune-rate 0.9 --prune-imp L2 \
--epochs 300 --batch-size 128  --lr 0.2 --wd 1e-4 --nesterov --scheduler multistep --milestones 150 225 --gamma 0.1

dynamic_model_pruning_with_feedback's Issues

TypeError: Conv2d._conv_forward() missing 1 required positional argument: 'bias'

Hi
Running this script cause the error in line 26 mnn.py
"python main.py cifar10 --datapath DATAPATH -a resnet --layers 56 -C -g 0 --save prune.pth -P --prune-type unstructured --prune-freq 16 --prune-rate 0.9 --prune-imp L2 --epochs 10 --batch-size 80 --lr 0.2 --wd 1e-4 --nesterov --scheduler multistep --milestones 150 225 --gamma 0.1"

" Conv2d._conv_forward() missing 1 required positional argument: 'bias'"

AttributeError

Hi
When I run your script, after validation, I encounter with
"AttributeError: 'Namespace' object has no attribute 'ensemble'
How can I resolve it?
Line 305 in main.py

Should the weights be masked at forward() ?

Hi, thank you for sharing your implementation.
I have noticed that the weights are masked out at every forwarding step such as in line 146 in models/resnet.py:
self.conv1.weight.data = torch.mul(self.conv1.weight, self.mask1.weight)

From my understanding of the paper, it seems the weights are not masked out to prevent prematurely pruning weights, which is also reflected in their algorithm in Appendix A.

I see that your implementation closely reproduces their result. Would you kindly give me some intuition on why the weights should be masked at every forward step?

Thanks in advance.