lapack exception (2) backward qr diff about backwardslinalg.jl

[maartenvd]

Ok so the problem is that my R is rank-defficient, and the inverse (which is done using trtrs) fails. I made sure my R's are now full rank and everything appears to work (as in, I now get different errors probably unrelated to backwardslinalg)

[GiggleLiu]

Thanks for you feedback. Yes, that happens sometime. LinearAlgebra backward functions have the problem of exploding gradients.

Now I am finding an approach to solve this problem one for all. The instruction level automatic differentiation on a reversible eDSL NiLang. Here is the code for a naive implementation of QR (without re-orthogonalization)

https://github.com/GiggleLiu/NiLang.jl/blob/master/project/qr.jl

This is in research stage and is not ready for productivity (it does not use BLAS!). Hopefully it can solve the gradient exploding problem in the future.

[maartenvd]

LinearAlgebra backward functions have the problem of exploding gradients.

When do they have this problem? I know svd has it, but I assumed that the backward differentiation of qr was stable (because the one inverse step is well behaved as long as R is well behaved).

How does nilang address this issue? I thought the exploding gradients had their origin in the inversion steps and I don't understand how you can prevent this.

[GiggleLiu]

svd, symeig have the spectrum degeneracy problem. QR (especially for rectangular matrices) has the rank deficiency problem in this implementation.

NiLang differentiate basic components, the floating point operations, rather than using manually derived formulas.

It depends on how forward program works. Like the Jacobi method in
https://web.stanford.edu/class/cme335/lecture7.pdf
Differentiating each instruction faithfully does not have a know caveat for now. Not sure it can solve the problem.