soumith / net2net.torch

Implementation of http://arxiv.org/abs/1511.05641 that lets one build a larger net starting from a smaller one.

Torch implementation of Net2Net: Accelerating Learning via Knowledge Transfer by Chen, Goodfellow, Shlens

• Proof of concept with unit tests
• Handles batchnorm layers in conjunction with linear and convolutional layers

n2n = require 'net2net'

-- net  = network
-- pos1 = position at which one has to widen the output
-- pos2 = position at which the next weight layer is present
-- newWidth   = new width of the layer
-- batchnorm layer should be between pos1 and pos2
-- batchnorm layer is modified to maintain identity-preserving mapping
n2n.wider(net, pos1, pos2, newWidth)

-- pos = position at which the layer has to be deepened
-- nonlin = type of non-linearity to insert
-- bnormFlag = boolean flag to insert batchnorm layer before the non-linearity
-- inserted batchnorm layer maintains identity-preserving mapping
-- make a forward pass through the model before calling n2n.deeper so that batch mean and variance can be computed
n2n.deeper(net, pos, nonlin, bnormFlag)

Example usage in test.lua

net2net.torch's Issues

Gradients Vanish

Hi,

I tried using net2net to expand a simple linear model. However, I've found the gradients to vanish at the point of expansion, causing the model to cease learning. Any ideas?

Addition of noise to break symmetry

Hi,
When you replicate the randomly chosen units, it seems that you don't add noise to break symmetry after the replication? If so, I think this may lead to not actually increasing the network capacity.