Issue loading models quantised with older GPTQ code: `ValueError: QuantLinear() does not have a parameter or a buffer named bias.` about autogptq

[TheBloke]

This seems to be resolved by #43 . I'll close it tomorrow once I've double checked a few more old models

[SirWaffle]

I'm seeing this issue with many models as well, loading models directly using auto-gptq ( not through the textgen webui )

[PanQiWei]

@SirWaffle Hi, can you try the main branch and see if this problem still exists?

[TheBloke]

And set strict=False in from_pretrained() on any model that throws this error

[SirWaffle]

using the latest from the main branch, I fail to import due to missing triton - I'm on windows. The previous version didn't require titon, only spit out a message if I did not have it installed. I will spend some time and see if I can get Triton* compiled manually for windows and setup to see if this addresses the issue

[Interpause]

perhaps, as a temporary mitigation an alternate load_quantize function should be written. The function would auto-detect the basename based on common model suffixes and guess the quantize_config from the model/folder name. The function could be marked as deprecated with instructions on how to convert old format models to new without needing to re-quantize.

[TheBloke]

Hi @Interpause

Unfortunately quantize_config doesn't help us: two models can have identical quantization config and one will work and one will not.

This is because there was a silent change in the GPTQ format. Models produced with older GPTQ-for-LLaMa code will throw the error described. There is no obvious indication as to whether a model will fail or not.

But we do already have a solution for this - we can pass strict=False to AutoGPTQForCausalLM.from_quantized() and then both old and new format models will load. That is the current recommended solution for this issue.

We have been discussing whether strict=False should therefore be default. I think perhaps it should be, although PanQiWei has been explaining that it might have implications in multi-GPU inference scenarios. So that needs to be tested.

[SirWaffle]

Im having a bit of an issue getting triton to work in windows, is there any hacky thing i can do to remove the reliance on triton from the main branch? or is triton now a required package that cant be detangled easily?

[SirWaffle]

using the change here: #85
I can confirm the older models load with strict=false, without triton issues under windows

[cxfcxf]

i can confirm the new change merged in fixed issue with strict=false setting on model load, i would say set it as default, most of my gptq model would simply not load without it

[Interpause]

What if the model name & version could be specified within quantize_config.json? The first would solve auto-loading models with custom names, and the second could be used to automatically determine if strict=False or other compatibility measures are needed. Maybe using version:-1 for GPTQ-for-Llama quantized models to automatically turn on strict=False.