Kernel verification failure during execution of axpby with complex float inputs about onemkl

[mmeterel]

@maleadt This looks like a driver issue or driver mismatch with the base tool kit to me. Are you using same driver version for 2024.0 and 2024.1?

I see this line: Is this the version used for both base tool kits?
Driver Version 24.09.028717

Can you try a newer driver with 2024.1?
Can you also try using Level-zero backend?

[maleadt]

Are you using same driver version for 2024.0 and 2024.1?

Yes, we build the driver separately from the MKL bits we use from Conda. Are there driver requirements on MLK from 2024.1? If so, are these documented anywhere?

I see this line: Is this the version used for both base tool kits?
Driver Version 24.09.028717

Correct; that's what I meant with NEO v24.9.28717 with IGC v1.0.16238.
This is also the latest driver available on https://github.com/intel/compute-runtime/releases, so which one exactly do you want me to try?

Can you also try using Level-zero backend?

How do I do that?

[eero-t]

Can you also try using Level-zero backend?

How do I do that?

NEO/IGC/SyCL is level-zero backend, but as you're selecting GPU, maybe that question was about whether it happens also when selecting CPU backend (with same NEO/IGC version)?

[maleadt]

whether it happens also when selecting CPU backend (with same NEO/IGC version)

I'm confused; does NEO/IGC have a cpu back-end? FWIW, using sycl::cpu_selector simply throws a PI_ERROR_DEVICE_NOT_FOUND.

In any case, upgrading to NEO 24.13.29138.7 seems to have fixed this.

[maleadt]

Scratch that, I'm still seeing the error with axpby and trmv: https://buildkite.com/julialang/oneapi-dot-jl/builds/1062#018f80f4-a375-44b0-9a4b-8a45ce9fb9e7

It doesn't reproduce consistently though; is it possible the validation isn't deterministic, or that compiled kernels are cached somewhere?

[eero-t]

I'm confused; does NEO/IGC have a cpu back-end? FWIW, using sycl::cpu_selector simply throws a PI_ERROR_DEVICE_NOT_FOUND.

Sorry, I was confused myself...

It doesn't reproduce consistently though; is it possible the validation isn't deterministic, or that compiled kernels are cached somewhere?

One way to check that it's always the same kernel code, could be dumping it on disk:
https://github.com/intel/intel-graphics-compiler/blob/master/documentation/shader_dumps_instruction.md