3-bit quantization fails during the packing stage about gptq-for-llama

[dustydecapod]

this impacts both 4-bit and 3-bit quantization

[dustydecapod]

I was able to fix this by removing line 142 from file quant.py, which sets self.bias. self.bias isn't even used in that function, so this line is unnecessary, and bias is None on pretty much every layer for this model.

[qwopqwop200]

This is because llama uses a linear layer with bias set to False.
This is an easily fixable problem by changing that line as you said.

[qwopqwop200]

i solved it.

@qwopqwop200 sorry for pinging you here, but I just wanted to ask - how much VRAM would a 3-bit version of LLAMA-30B require (e.g. can it run on 16GB VRAM), and would it be fast enough (like at least a few tokens per second) on e.g. RTX A4000?

[qwopqwop200]

If you do 3-bit quantization, it seems to consume about 14gb of memory at 30B. The inference rate is expected to be less than 0.1s/token.

@qwopqwop200 wow, thanks for all of your efforts, that's amazing! I can tolerate accuracy loss as it'll still be much better than 4-bit LLAMA 13b.