The installed version of bitsandbytes was compiled without GPU support. 8-bit optimizers, 8-bit multiplication, and GPU quantization are unavailable. about deepke

[guihonghao]

你好，环境版本是下面这些。如果8bit量化存在问题，可以尝试使用4bit量化。

accelerate==0.21.0
transformers==4.33.0
bitsandbytes==0.39.1

[jiangweiatgithub]

多谢回复。4bit量化，该如何修改代码？

[guihonghao]

quantization_config=BitsAndBytesConfig(     
    load_in_4bit=True,
    llm_int8_threshold=6.0,
    llm_int8_has_fp16_weight=False,
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
)

model = AutoModelForCausalLM.from_pretrained(
    model_path,
    config=config,
    device_map="auto",  
    quantization_config=quantization_config,
    torch_dtype=torch.bfloat16,
    trust_remote_code=True,
)

设置并传入quantization_config参数

[zxlzr]

请问您还有其他问题吗？