[Bug]: ValueError: The quantization method fp8 is not supported for the current GPU. Minimum capability: 90. Current capability: 86 about vllm

[agt]

Hi @Maydaytyh

GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
[...]
ValueError: The quantization method fp8 is not supported for the current GPU. Minimum capability: 90. Current capability: 86.

FP8 is only supported on >=sm90 i.e. Hopper cards. (Per fp8.py, support for sm89 (Ada, 4090) may come once vLLM upgrades to Pytorch 2.3.0)

AWQ and GPTQ quantization are much less hardware-specific, you might try using those.

[seungyeon-seo]

I have same error.

ERROR 04-24 21:28:44 worker_base.py:157] KeyError: 'model.layers.55.mlp.down_proj.in_scale'
KeyError: 'model.layers.55.mlp.down_proj.in_scale'
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] Error executing method load_model. This might cause deadlock in distributed execution.
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] Traceback (most recent call last):
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] File "/home/d/anaconda3/envs/3.8/lib/python3.8/site-packages/vllm/worker/worker_base.py", line 149, in execute_method
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] return executor(*args, **kwargs)
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] File "/home/d/anaconda3/envs/3.8/lib/python3.8/site-packages/vllm/worker/worker.py", line 117, in load_model
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] self.model_runner.load_model()
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] File "/home/d/anaconda3/envs/3.8/lib/python3.8/site-packages/vllm/worker/model_runner.py", line 162, in load_model
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] self.model = get_model(
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] File "/home/d/anaconda3/envs/3.8/lib/python3.8/site-packages/vllm/model_executor/model_loader/init.py", line 19, in get_model
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] return loader.load_model(model_config=model_config,
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] File "/home/d/anaconda3/envs/3.8/lib/python3.8/site-packages/vllm/model_executor/model_loader/loader.py", line 224, in load_model
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] model.load_weights(
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] File "/home/d/anaconda3/envs/3.8/lib/python3.8/site-packages/vllm/model_executor/models/llama.py", line 411, in load_weights
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] param = params_dict[name]
(RayWorkerWrapper pid=3766121) ERROR 04-24 21:28:45 worker_base.py:157] KeyError: 'model.layers.55.mlp.down_proj.in_scale'

[mgoin]

Yes this is intentional, at the moment FP8 will only be supported where we have native hardware support.