Comments (3)
Hi, sorry for delayed response. In my experience SWA works best with SGD. Adam sets the learning rates adaptively, which is not ideal for SWA. However, we did see some improvement with other optimizers as well. I recommend trying to tune the learning rate schedule (try increasing the learning rates during the SWA stage), or maybe switching to SGD for the SWA stage.
from swa.
As I see in TensorFlow Adam have trainable parameters, so the question is should we exclude these parameters from averaging? Same question for BN trainable parameters.
from swa.
Hey @mrgloom. The adam parameters and BN parameters are not trainable parameters of the network. In fact, the former are tensors stored in the optimizer state, and the latter are buffers of the model. They should not be averaged. However, you need to fix the batchnorm statistics for the SWA model in the end of training (https://pytorch.org/blog/stochastic-weight-averaging-in-pytorch/#batch-normalization)
from swa.
Related Issues (20)
- About bn_update HOT 5
- About conv_init(m)
- CUDA out of memory HOT 1
- Unable to re-produce SGD numbers HOT 7
- [ Question ] Isn't the mid-training evaluation of the SWA performance corrupting the batchnorm running averages ? HOT 4
- SWA compared to Exponential Moving Average (EMA) HOT 4
- Codes for figure 1,3,4, and 5 HOT 1
- Isn't the weight learnt from each minibatchs are supposed to be averaged? HOT 5
- release code for training imagenet HOT 1
- Finetuning model with SWA HOT 1
- How many weights instances stored in memory simultaneously? HOT 1
- SWA with distributed training HOT 1
- SWA with Torchbearer HOT 3
- Figure 3 plot HOT 1
- About preresnet results HOT 2
- How to reproduce result of Fig.1 in the paper? HOT 5
- About calculating parameters of BN HOT 1
- performance drop due to batch norm params recalculation HOT 1
- How about sgdr? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from swa.