[Doc]: Why is the PA kernel time cost in the decode phase optimized after turning on Prefix Caching? about vllm

[robertgshaw2-neuralmagic]

APC improves TPOT because there is less overhead on the system from processing prefills.

vllm has a central LLMEngine, which runs a step each timestep. A step can be either a prefill or a decode. So a good mental model for vllm is that it is running decodes for all active requests constantly with "pauses" to process prefills from new requests to add them to the batch.

As a result, TPOT = n_tokens_generated * time_per_decode_step + n_prefills_processed * time_per_prefill_step

Since APC reduces time_per_prefill_step TPOT is reduced. Additionally, APC indirectly reduces (a) n_prefills_processed and time_per_decode_step because it reduces E2E latency and therefore (a) there are on average less prefills to process while a specific request is running and (b) the average batch size is lower.

[wjj19950828]

APC improves TPOT because there is less overhead on the system from processing prefills.

vllm has a central LLMEngine, which runs a step each timestep. A step can be either a prefill or a decode. So a good mental model for vllm is that it is running decodes for all active requests constantly with "pauses" to process prefills from new requests to add them to the batch.

As a result, TPOT = n_tokens_generated * time_per_decode_step + n_prefills_processed * time_per_prefill_step

Since APC reduces time_per_prefill_step TPOT is reduced. Additionally, APC indirectly reduces (a) n_prefills_processed and tie_per_decode_step because it reduces E2E latency and therefore (a) there are on average less prefills to process while a specific request is running and (b) the average batch size is lower.

@robertgshaw2-neuralmagic Thank you for your reply~

But through nsys analysis, I saw that in the decoding stage, only the page attention kernel has reduced time consumption, and other kernels are close. Is this reasonable? Why?

kernel name:
void vllm::paged_attention_v1_kernel<unsigned short, unsigned short, (int)128, (int)16, (int)128, (bool)0>(T1 *, const T1 *, const T2 *, const T2 *, int, float, const int *, const int *, int, const float *, int, int, int, float)

with APC vs without APC time cost:
252.511 us vs 382.047 us
13b model , 300 input_len / 20 output_len, bs 100 in A100 TP1