max_grad_norm is ignored in FP16 training about deepspeed

[LvanderGoten]

I just saw that there is a "gradient_clipping" option in the config. However, setting it to 1.0, does not eliminate the gradient overflow messages. Is this expected?

[tjruwase]

Gradient overflow messages, typically at the beginning of training, are not surprising. But this should eventually stop once the dynamic loss scale has decreased to a point that allows training stability.

Are you not seeing the overflow messages stop?

[LiweiPeng]

Regarding to the original question, what's the purpose to disable 'max_grad_norm' for fp16?
if self.fp16_enabled() and 'max_grad_norm' in optimizer_parameters.keys():
optimizer_parameters['max_grad_norm'] = 0.0

[tjruwase]

The rationale for this was to support both the old and new ways of specifying gradient clipping. The difference is that in the old way, gradient clipping is specified as max_grad_norm parameter of the fp32 optimizer, while in the new (and more intuitive way IMHO) gradient clipping is handled in the fp16 wrapper optimizer, such as here. In summary, this logic allows the fp16 optimizers in DeepSpeed to handle both old and new gradient clipping styles.

All that said, we are thinking of ways to improve the interface and avoid this confusion, and we certainly appreciate your input on that.