Should lower/optimize behave differently based on the compilation mode? about glow

[qcolombet]

Hi @nadavrot,

In essence, yes that's what I am suggesting with one difference.
Instead of creating a BN-train node that would replace the BN-inference during differentiation, I was thinking of replacing BN with two nodes BN + meanVarNorm during differentiation.

The effects are the same, the difference is that we wouldn't have to change the optimizations.
I.e.,
Input:
BN-inference

Differentation:

• Today
  BN-inference
  BN-gradient

• Your proposal
  BN-train
  BN-gradient

• My suggestion
  meanVarNorm
  BN-inference
  BN-gradient

Essentially, BN-train is equal to (BN-inference(meanVarNorm)).

[nadavrot]

@qcolombet Ah, got it. We can split BN-train into two sequential parts. I think that this proposal makes sense.

[qcolombet]

Hi @jfix71 - interested in doing this?
If not, assign it back to me.

[nadavrot]

Thanks for raising this issue @qcolombet. Let's talk about BatchNormalization. BN is implemented with conceptual 3 nodes: BN-inference, BN-train and BN-gradient. If I understand your suggestion, when we construct the inference graph (when loading C2 models, or when constructing our unit tests) we'll emit the BN-inference node. Later, during differentiation we'll insert the BN-gradient node, just like we do today, but also convert BN-inference into BN-train. Also, we would change the BN optimization to work on BN-inference and not on BN-train. Is this what you are suggesting? Generally, I like this direction. I think that it makes sense.

[jfix71]

@qcolombet I'll take a look today!