Deep Q Controller Possible Error about tensorflow-deepq

[siemanko]

Thanks for investigating this!

Because of the fast the temp_diff is squared later this should not matter, i.e. the gradient ends up being the same for self.value_given_action regardless of where it is in tempdiff:

e = (a-b)^2
de/da = 2(a-b) * d(a-b)/da = 2 (a-b)

or if you swap

e = (b-a)^2
de/da = 2(b-a) * d(b-a)/da = 2(a-b) * -1 = 2(a-b)

so the gradients come out being the same. Unless TF is doing something crazy. But I am reluctant to believe that TF would do something that fundamentally wrong.

Can you provide a reproducible experiment where one version consistently outperforms the other for a couple of different random seeds?

[Mazecreator]

Hummm... Sorry, I switched it back and cannot reproduce the problem. My model must have been underfit originally and I must have modified it around the same time I tried changing the sign. I was surprised as well, but I figured it may have something to do with the Stop Gradient command causing the error indirectly.