Comments (3)
Hey @seyoulala thanks for your kind words! :)
I see a number of ways of doing it:
- Using multiple trainers: to each trainer you can pass a
lr_scheduler
(see here). Therefore you could do something like
model = WideDeep(...)
first_scheduler = torch.optim.lr_scheduler.StepLR(...)
warmup_trainer = Trainer(model, objective, lr_schedulers=first_scheduler)
# At this point the weights of the model are "warm"
trainer = Trainer(model, objective, ...). # with a 2nd scheduler if you wanted
-
Passing a Sequential LR scheduler: in principle, the
Trainer
accepts any valid pytorch scheduler. Therefore, you could build your own sequential learning rate scheduler and pass it. If this throws an error let me know, because it would be a bug to fix. Personally, I have never tried it (but I will now :) ) -
And the last one, the
fit
method allows for a series of warm up parameters 😉. At the moment I implemented two routines. Have a look here, to thefinetune
parameter, aliased aswarmup
. Also to this example notebook.
Let me know if you need any more info/help
from pytorch-widedeep.
thanks !build my own learning rate scheduler by SequentialLR of pytorch can deal this problem
from pytorch-widedeep.
okay, thanks @seyoulala
from pytorch-widedeep.
Related Issues (20)
- Image Preprocessing takes a lot of time HOT 2
- Not Being able to reproduce Bert results HOT 5
- pytorch vision module error HOT 1
- save_best_only error and NaN during training HOT 9
- CyclicLR throws ZeroDivisionError when finetuning with a single batch. HOT 2
- EarlyStopping does not store and restore the model HOT 5
- Can I use time series data HOT 6
- CUDA error: device-side assert triggered HOT 5
- Wrong paper links on ContrastiveDenoisingTrainer HOT 2
- how to save the best Epoch HOT 11
- Dropout layer being created on forward pass (in MultiHeadedAttention) HOT 1
- about Wide's input dim HOT 5
- ImportError: cannot import name 'LRScheduler' from 'torch.optim.lr_scheduler' HOT 8
- OSError when importing the package HOT 4
- AttributeError: 'TabMlp' object has no attribute 'with_fds' HOT 3
- Colab session crash on .fit HOT 3
- IndexError: index out of range in self HOT 4
- Problems running transformer models HOT 5
- 'TextPreprocessor' object has no attribute 'embedding_matrix' HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytorch-widedeep.