lefteyex / warmup

A Warmup Scheduler for Pytorch to achieve the warmup learning rate at the beginning of training.

👋 Hi, LEFTeyex here

• 😄 I’m currently studying for a PhD.
• 🌱 I focus on CV and AI.

warmup's Issues

more description about the warmup behaviour

I want to ask about the warmup behavior. For example, I use StepLR with step_size=5, gamma=0.1, start_lr=0.3, len(dataloader)=100. If I set the warmup_step=800, what is the learning rate after warmup_step? is it 0.3 or 0.03?

torch lr_schedulers not accepted

When instantiating the object WarmUpScheduler and passing e.g. the torch.optim.lr_scheduler.CosineAnnealingLR as in example.py execution fails due to the following error:

Traceback (most recent call last):
  File "test.py", line 26, in <module>
    warmup_scheduler = WarmUpScheduler(optimizer, lr_scheduler,
  File "/Users/tomas.gadea/Desktop/test/env/lib/python3.8/site-packages/warmup_scheduler_pytorch/warmup_module.py", line 52, in __init__
    raise TypeError(f'{type(lr_scheduler).__name__} is not a lr_scheduler in pytorch')
TypeError: CosineAnnealingLR is not a lr_scheduler in pytorch

It does not work for any of the typical torch schedulers.