PyTorch 2.2 pre-built wheel about autogptq

[Andrei-Aksionov]

Thanks a lot @fxmarty.

I only noticed that the package with Marlin kernel is installed even on a system that doesn't support it, e.g. T4. Which is fine. But I would like that a user is notified about this incompatibility.
The error appears only when I try to run forward pass, not during layer initialization. (And the error message doesn't tell me what is exactly wrong.)

import torch
from auto_gptq.nn_modules.qlinear.qlinear_marlin import QuantLinear

x = torch.rand((1, 1, 128), device="cuda").half()
layer = QuantLinear(4, 128, 128, 256, False).to("cuda")
layer(x)
>> 
RuntimeError: CUDA error: invalid argument
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

So perhaps it's better to add a check inside init method, something like this:

import subprocess

output = subprocess.check_output("nvidia-smi --query-gpu=compute_cap --format=csv,noheader".split())
compute = float(output.strip().decode())
if compute < 8.0:
    raise NotImplementedError("[Error message]")

Or maybe I misunderstood something? 🤷‍♂️

[fxmarty]

Good call! Currently the check is in from_quantized

AutoGPTQ/auto_gptq/modeling/_base.py

Lines 1225 to 1226 in 74212f5

	if not torch.cuda.get_device_capability()[0] >= 8:
	raise ValueError(f'Can not use Marlin int4*fp16 kernel with a device of compute capability {torch.cuda.get_device_capability()}, the minimum compute capability is 8.0 for Marlin kernel. Please do not use `use_marlin=True`, or please upgrade your GPU ("The more you buy, the more you save." - Taiwanese proverb).')

but it should probably be in marlin's QuantLinear init.

[Andrei-Aksionov]

I knew that there should be an easier way to check compute capability 🙂.

Yes, in my opinion, this check belongs to the QuantLinear class.

"The more you buy, the more you save." - Taiwanese proverb

Now I know :)

[fxmarty]

Thank you. I am planning to make a release hopefully this week, with Marlin kernel & built against PyTorch 2.2. Still need to add guards against __CUDA_ARCH__ in marlin codebase as it can only be compiled for compute capability >=8.0.

I'll add a fast repacking first as well.

[Andrei-Aksionov]

Awesome news!
Thanks a lot 🤗

[fxmarty]

Hi @Andrei-Aksionov, AutoGPTQ 0.7.0 is released, check out https://github.com/AutoGPTQ/AutoGPTQ/releases/tag/v0.7.0 & https://github.com/AutoGPTQ/AutoGPTQ?tab=readme-ov-file#installation!

[fxmarty]

@Andrei-Aksionov Added a check in #567, I'll likely do a patch this week.

[Andrei-Aksionov]

Thanks!