Since cudarc 0.11.4 error with PTX -- CUDA_ERROR_UNSUPPORTED_PTX_VERSION about candle

[LaurentMazare]

You may want to try removing your target directory or do some form of cargo clean to trigger a new compilation from scratch (the new cudarc might have picked a different cuda version on your system and you would need to recompile the PTX files).

[CoffeeVampir3]

You may want to try removing your target directory or do some form of cargo clean to trigger a new compilation from scratch (the new cudarc might have picked a different cuda version on your system and you would need to recompile the PTX files).

Cargo clean, deleting lock file, or even making an entirely new cargo project (including deleting rust entirely and reinstalling) don't have any change in outcome for me. Any new candle projects give me the same error.

From the following steps:

rustup self uninstall
yay -Rdd rustup
yay -S rustup
rustup toolchain install nightly
rustup default nightly
cargo clean
rm Cargo.lock
rm -rf target
cargo build

https://www.toptal.com/developers/hastebin

-- Edit:
Seems deleting ~/.nv and the entire ~/.cache does not change anything either, along with the above steps. The quantized example from candle also doesn't run when using a cuda device (same error)

-- Edit 2:
There's also some new metal kernels around the same time specifically dealing with u32_f32 including tests, unsure if these have snuck in for non-metal.

[CoffeeVampir3]

Okay! After a few days of troubleshooting I've tracked down the problem. This wasn't clear to me, but the issue was a cuda 12.5 VS my driver supported cuda 12.4 mismatch. Updating to cuda 12.5 compatible drivers fixes this problem. So the issue was really not the PTX version at all, but the cuda minor version mismatch.