cuda extension problem about gptq-for-llama

[qwopqwop200]

I'm not familiar with pytorch cuda extension.
However, this code doesn't work.(RTX3090)
Also, please explain the benefits of this change.

[WuNein]

I tested on my rtx 3090 and 4090, in nvidia docker. IT is a must setup.

[qwopqwop200]

nvcc_args = [
    '-gencode', 'arch=compute_80,code=sm_80',
    '-gencode', 'arch=compute_86,code=sm_86',
]

I have to use this code to make it work normally.
Does the CUDA version have any effect?

[WuNein]

sm_90 is H100, 80-86 shall be 3090 - 4090.
I have to specific version, or it will generate all sm, include parcel ones.
This shall be notify in README, i think.

For your problem, i think it is related to driver version, older drivers do not have sm_90.

[qwopqwop200]

It seems to depend on the cuda version. sm90 is supported from cuda 12.0.

https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

[qwopqwop200]

We currently use Triton instead of CUDA.