AWQ inference is 50% slower than GPTQ about autoawq

[casper-hansen]

How did you measure performance? This is much lower than my own benchmarks which you can see in the README.

[gestalt73]

Hey @casper-hansen for these metrics I took a shortcut and ran them using text-generation-webui.

If I can find the time today I'll make a second attempt using a plain python script.

[casper-hansen]

You should try the benchmark script that I included in examples. GEMV achieves 90% of ExLlamaV2 speed and GEMM achieves 70%.

Also do note that AutoAWQ is a more broad framework. We aim for high speed but make compromises that does ExLlama does not.

[gestalt73]

I cobbled together a version of the script to gauge performance and you're absolutely right, the implementation in text-generation-webui is slowing things down.

I can confirm that for the first run AutoAWQ approaches 90% of ExLlamaV2 performance at least for TheBloke/Llama-2-7b-Chat-AWQ.

What interesting is that the inference performance goes down for every run. This was run against the standard 0.1.0 version so it errored out after the last run shown.

(textgen) alansrobotlab@goliath:~/Documents/brayden/langchain$ python testawq.py
Fetching 13 files: 100%|██████████████████████████████████████████████████████████████████| 13/13 [00:00<00:00, 226248.76it/s]
Replacing layers...: 100%|████████████████████████████████████████████████████████████████████| 32/32 [00:01<00:00, 20.25it/s]
Tokens per second:   120.23952638403186
Tokens per second:   112.92177108754163
Tokens per second:   103.59184799892934
Tokens per second:   95.48121527004926
Tokens per second:   99.72065805469065
Tokens per second:   98.25381852653379
Tokens per second:   94.36816405132929
Tokens per second:   90.7243856285012
Tokens per second:   88.71160320470709

from awq import AutoAWQForCausalLM
from transformers import AutoTokenizer
import time

#model_name_or_path = "TheBloke/CodeLlama-34B-AWQ"
model_name_or_path = "TheBloke/Llama-2-7b-Chat-AWQ"
#model_name_or_path = "casperhansen/vicuna-7b-v1.5-awq"

# Load model
model = AutoAWQForCausalLM.from_quantized(
    model_name_or_path,
    #"awq_model_w4_g128.pt",
    fuse_layers=True,
    trust_remote_code=False,
    safetensors=True,
    max_new_tokens=4096,
)
tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, trust_remote_code=False)

tokens = tokenizer(
    "# What are all fo the different quarks in nuclear physics and when and where were they discovered?\n\n", return_tensors=>
).to("cuda")

# Generate output
for _ in range(10):
    start = time.time()
    generation_output = model.generate(
        **tokens,
        do_sample=True,
        temperature=0.2,
        top_p=0.95,
        top_k=0,
        max_new_tokens=512,
    )
    elapsed = time.time() - start
    count = len(generation_output[0])
    #print("Elapsed:  ",elapsed)
    #print("Tokens:  ",count)
    print("Tokens per second:  ",(count/elapsed))
    #print(tokenizer.decode(generation_output[0], skip_special_tokens=True))