RuntimeError: CUDA error: no kernel image is available for execution on the device on 3xP40 about mamba

[Tworan]

I came into the same issue with gtx1080ti which compute capability is compute_60. In the setup.py file, I see both the mamba_ssm and causal_conv1d only compile the code with compute_70, compute_80 and compute_90 except compute_60 which is corresponding to the sm_60:

mamba/setup.py

Lines 108 to 114 in 86a3a90

	cc_flag.append("-gencode")
	cc_flag.append("arch=compute_70,code=sm_70")
	cc_flag.append("-gencode")
	cc_flag.append("arch=compute_80,code=sm_80")
	if bare_metal_version >= Version("11.8"):
	cc_flag.append("-gencode")
	cc_flag.append("arch=compute_90,code=sm_90")

So I add two line to both setup.py file of mamba_ssm and causal_conv1d, then reinstall both packages and it works for me.

    #add
    cc_flag.append("-gencode")
    cc_flag.append("arch=compute_60,code=sm_60")
    # 
    cc_flag.append("-gencode")
    cc_flag.append("arch=compute_70,code=sm_70") 
    cc_flag.append("-gencode") 
    cc_flag.append("arch=compute_80,code=sm_80") 
    if bare_metal_version >= Version("11.8"): 
        cc_flag.append("-gencode") 
        cc_flag.append("arch=compute_90,code=sm_90")

And I think this may be because the Pascal architecture GPU(compute_60) does not support the bf16 data type. And this may cause the functions using bf16 to be unusable.

[hhhhpaaa]

I built a wheel under the Python 3.10, PyTorch 2.1, and CUDA 11.8 environment to support the compute_60. For details, please refer to the link.
@SamsongB @DewEfresh

[masc-it]

Have you tried running it on a single gpu? (Maybe a smaller one) If that's the case, I think there are some issues with the custom operators and DDP.

Or, it could be the fact that P40 is missing some kernels that the mamba optimized version uses. On my ampere-series card no probs whatsoever.

[SamsongB]

I also see the same issue with GM200 [GeForce GTX TITAN X], tried the above solution with compute_60, but still cannot seem to run. is there an update regarding this?

[a987042035]

@DewEfresh Hello, my dear. Have you solved your problem? The single card p40 I use also has this problem. Please help me with my cuda version 11.8