Title pretty much sums it up; having wrappers for the cuBLAS and cuSOLVER libraries would likely be extremely useful. Will likely require a few iterations to find the "right" level of abstraction though.
It would be great to simplify the handle management, workspace calculations, as well as function invocations. The latter might prove to be a bit too much, in which case error handling functions for said invocations could turn out to be Good Enough™.
I think it would be beneficial to investigate various possible RAII techniques to simplify device memory management. The immediate possibilities that spring to mind is either to create custom pointers that wrap device pointers, perhaps along the lines of std::shared_ptr or similar. Another possibility is to simplify this in the case of STL containers via implementing a custom allocator.
However, the ideal approach is likely to do both (or something in between if that even makes sense).
TL; DR I want to completely remove the need to manually invoke cuda::free.