`.to` is not supported for HQQ-quantized models about hqq

[Abdullah-kwl]

check out my Notebook link: https://drive.google.com/file/d/1Oq371TVP2QMQqsw0juYwt4-T1z9fvGyC/view?usp=sharing

i want to use free Colab as I use during BnB quantization

[mobicham]

Hi, the branch and the example: https://gist.github.com/mobicham/cb07c1eff443ad0918c49ab7bb03e269 are working fine, I just tried it now. Make sure:

• You have a clean installation of transformers from https://github.com/mobiusml/transformers/tree/stable (not main)
• Make sure the GPU is selected in the Colab session

[Abdullah-kwl]

I used the same transformer from the stable branch https://github.com/mobiusml/transformers/tree/stable, but facing the same issue ,
I am also using GPU ,

Please Review my code if there is any mistake:

#first download the cloned repo of transformer from stable branch https://github.com/mobiusml/transformers/tree/stable
#download hqq from master branch repo https://github.com/mobiusml/hqq

---

!pip install git+https://github.com/Abdullah-kwl/Transformers-HQQ-Integration.git

!pip install git+https://github.com/mobiusml/hqq.git

import torch, transformers

from hqq.core.quantize import *

model_id = "senseable/WestLake-7B-v2"

#Basic

quant_config = BaseQuantizeConfig(nbits=4, group_size=64)

cache_path = ''

compute_dtype = torch.float16

device = 'cuda:0'

#pass BaseQuantizeConfig to AutoModelForCausalLM

model = transformers.AutoModelForCausalLM.from_pretrained(model_id, cache_dir=cache_path, torch_dtype=compute_dtype, device_map=device, quantization_config=transformers.HQQConfig(quant_config) )

tokenizer = transformers.AutoTokenizer.from_pretrained(model_id)

---

After this cell run it do every thing but in last it show the error ValueError: .to is not supported for HQQ-quantized models.
.to is not supported for HQQ-quantized models
#39

please look at https://github.com/mobiusml/transformers/blob/stable/src/transformers/modeling_utils.py
this is from transformer stabel branch

[mobicham]

Strange, I tried the same branch and it worked, with Llama2-7B. Can you with Llama2-7B?

[mobicham]

Try using HQQ with transformers: huggingface/transformers#29637