Prefix Caching Does Not Require Prompt Template and Other Ambiguities about tensorrt-llm

[thorjohnsen]

If you have a prompt structured as "Please ask [user] how their day is going", and [user] changes every request, only the tokens "Please ask" will potentially be reused. This is subject to kv cache block size. Default block size is 64, but can be any power of two <= 128. In order to reuse a block, that block must be full and all tokens in the block must be a perfect match.

[thorjohnsen]

I am unaware of any feature that allows client to reserve cache pages to permanently store a set of predefined shared prefixes. Regular prefix caching will work for any shared prefix, but is subject to a few limitations that might limit how much benefit you see:

1. Only full blocks that are a perfect match will be reused. For instance, if block size is 64 and your shared prefix has 180 tokens, only the first 128 tokens will be reused.
2. Reusable KV cache blocks are evicted when the memory is needed for other purposes. On a system with high load, eviction is quite likely to occur, especially if some time passes between each use of the shared prefix.

[thorjohnsen]

We can add a feature to reserve memory for shared prefix if client(s) request it. It would be added as one more input, probably named "prefix_length" that is an int32 value that says that the first "prefix_length" tokens of the request are part of a shared prefix. This would have to be an "advisory" field, so we would reserve space for the field if we can. Besides making the prefix more permanent by delaying eviction, we could remove the "full block" limitation so that the entire prefix could be shared, not just the part that fits into a whole number of blocks.

[ethan-digi]

@thorjohnsen thank you very much for your response. I think a prefix_length field, potentially with full block limitation removed, would be extremely useful. Essentially a way to indirectly sacrifice caching of blocks that are highly unlikely to be reused (e.g., end of the prompt) in order to have guaranteed caching of tokens that are known to be reused. I would like to assist with it but I don't think I have the time at the moment.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."