Comments (2)
As you can see, if the next layer and the current layer have different output shapes, you need to linearly project the output of the current layer so that it matches the dimensions of the output of the following layer.
The identity transform in Eq. 2 is the same as in Eq. 1, which is just the shortcut connection of the ResNet. For shortcuts changing the spatial size (dashed arrows in http://arxiv.org/abs/1512.03385, Fig 3), there are two options, explained in the first paragraph of page 4 of http://arxiv.org/abs/1512.03385. An implementation of that paper is given in https://github.com/Lasagne/Recipes/blob/master/papers/deep_residual_learning/Deep_Residual_Learning_CIFAR-10.py, including these two options.
On page 8 of the stochastic depth paper, they mention that for blocks changing the number of filters and spatial dimension, they "replace the identity connections in these blocks by an average pooling layer followed by zero paddings to match the dimensions." This is neither of the two options in the ResNet paper, but it's easy enough to modify the existing Lasagne Recipe to do so.
If in doubt about what they did in the stochastic depth paper, refer to the source code at https://github.com/yueatsprograms/Stochastic_Depth.
/edit: If you manage to reproduce the results of the stochastic depth paper (CIFAR-10 will be the easiest target), we'd appreciate a PR to this repository.
For additional fun, note that there's also a second ResNet paper (https://arxiv.org/abs/1603.05027) which was done concurrently to the stochastic depth paper. It's possible that combining these two would yield even better results. A Lasagne implementation is here: https://github.com/FlorianMuellerklein/Identity-Mapping-ResNet-Lasagne.
from recipes.
Ah, thank you! How did I not notice the implementation detail section in the paper... I'll take a crack at this and see what I can come up with!
from recipes.
Related Issues (20)
- 3D UNet implementation HOT 7
- reason behind low value of parameters in VGG19 HOT 1
- error when set values for vgg-19 HOT 2
- modelzoo resnet50.py incompatible to Python 3 HOT 3
- Implementation of Convolutional Spatial Transformer and Siamese network HOT 3
- no sandbox.cuda
- Bad argument to Theano: No. of dimensions changes in the error after reshaping
- Question not an Issue: fliping the arrays HOT 1
- cifar100 with resnet HOT 1
- pretrained network for small images HOT 1
- https://s3.amazonaws.com/lasagne/recipes/pretrained/imagenet/vgg16.pkl HOT 6
- vgg16.pkl without aws cli
- Need help with S3 Browser based downloads HOT 1
- Dice coeff is not changing since the first epoch and binary accuracy changes and is increased to 1?
- Problem with op.grad in OpFromGraph - Guided Backpropagation
- Wrong order of stride and pad arguments in build_simple_block HOT 3
- Broken links in Video features with C3D.ipynb example HOT 5
- Training C3D
- Wrong pretrained weights for UNet example HOT 1
- DICE coefficient loss function HOT 23
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from recipes.