Strange gradients. about neural_renderer

[nkolot]

I've located the source of assymetry. If you look at the lines (401,402) and (481,482) in this file, there is a statement

if (diff_grad <= 0)
 continue;

If you comment this out, then the 2 gradient values have the same magnitude and opposite signs. However I feel that this "hack" is not correct, even though it still passes all the test cases. Probably the code in this part needs to be written again from scratch. It is not documented well by the authors so it won't be trivial to figure out how it works.

Nevertheless, can you try to run it with this fix and see if your results improve?

[nkolot]

Ooops! That is indeed very weird. So it is an issue of the original Chainer implementation too. I will have to look into the equations of the paper and the CUDA code to try and figure out what is happening here.

We should expect that the gradients before and after should have the same magnitude and flipped sign.

[airalcorn2]

Thanks for looking into it, @nkolot. Initial results seem encouraging.

Positive Loss/Blacken
Before Fix

After Fix

Negative Loss/Whiten
Before Fix

After Fix

Agreed that the fix feels "hack-y", but it's hard to understand why the conditional is there in the first place because of the lack of documentation. My best guess is it's related to equations (1), (3), and (4) in the paper, where they set the partial derivative to zero when delta_P_j * delta_I_j >= zero, but the conditional in the code doesn't reference delta_P_j and is <= rather than >=. Very strange!

[nkolot]

I am reopening the issue, just to keep track that I have to properly fix this and submit a PR. Will do it when I find time, probably later in the week.