clubiedong / qaq-kvcachequantization Goto Github PK

in evaluator.py, Evaluator._evaluate_single func
you use the attention : question_attentions = [attn[:,:,:question_len,:question_len].to(self.device) for attn in result.attentions]
which is calculated by the first forward (without Quantize), not calculated by the quantized kv value

So I have some questions about this, and I don't know what difference it would make to replace it with the attention calculated using quantified kv values

Thanks for your good work!

Thanks for the work!

I notice that you quantize the kv cache only once during your experiments and pick the candidate choice with highest sum probabilities. I'm worried about the latency influence to general tasks like multi-round chat, do you have related experiments?

very very very good paper!

读起来让人心旷神怡!