DROCC Learning Rate Sheduler about edgeml

[mpierrau]

I just found the repo of one of the authors (Sachin Goyal), in which the following snippet could be found in cifar/code/train_cifar_pgd.py:

#LR SCHEDULER
def adjust_learning_rate(optimizer, epoch, args):
    """decrease the learning rate"""
    lr = args.lr
    if epoch <= args.epochs:
        lr = args.lr * 0.01
    if epoch <= 0.80*args.epochs:
        lr = args.lr * 0.1 
    if epoch <= 0.40*args.epochs:
        lr = args.lr    
    for param_group in optimizer.param_groups:
        param_group['lr'] = lr

which hints at the used learning rate scheduler for the CIFAR-10 experiment. The can easily be replaced by a torch.lr_scheduler.StepLR scheduler with step_size=0.4*n_epochs and gamma=0.1, where n_epochs is the total number of epochs.

However, the arguments for lr_scheduler in drocc_trainer.py suggests that the learning rate may be different for the "pretraining" (without adversarial loss) phase.

I suppose I can experiment a bit with it, but I definitely think that the lr_scheduler utilized in drocc_trainer.py should be either removed or properly explained, and I would much appreciate some clarification on the parameters used during the experiments.

Cheers!

[SachinG007]

Hi @mpierrau ,
Thanks for your interest in our work.

1)The LR scheduler used has been specified in the examples file.
For example : for CIFAR : here

2)We havent used the standard torch StepLR just for the simple reason that we didnt want to consider the initial "pretraining" warm up steps (without adversarial phase) as you correctly pointed out. The learning rate for pretraining is not different, but constant and equal to the specified initial LR.
I believe reading the code snippet linked above (linking here again) should clarify all your doubts.

Feel free to reach out for further queries.

Thanks
-Sachin