any explanation of final_lr ? about keras-adabound

[titu1994]

It's from the original implementation here - https://github.com/Luolc/AdaBound/blob/master/adabound/adabound.py

In essence, it's a bypass technique - to allow learning rate schedules to update the internal final_lr which the user cannot access.

[iperov]

my stable Adam lr is 5e-5
suggestion to lr and final_lr for AdaBound ?

[titu1994]

With an LR that low, the model must be learning very slowing.

In any case, Adabound's final_lr is the lr of "SGD with gradient clipping". The authors suggest 0.1 is fine for Adam lr of 1e-3. If you are scaling by 0.005 then perhaps scaling final lr by that much might be beneficial as well, though I can't say without tests.

The paper suggests that the learning rate of 0.1 for sgd is quite good, and that performance isnt heavily affected by minor changes to it. Scaling it 0.005 times however is a major change that is not explored in the paper if I remember, so I can't say what effect it will have on the model training.