Comments (5)
If you have a prompt structured as "Please ask [user] how their day is going", and [user] changes every request, only the tokens "Please ask" will potentially be reused. This is subject to kv cache block size. Default block size is 64, but can be any power of two <= 128. In order to reuse a block, that block must be full and all tokens in the block must be a perfect match.
from tensorrt-llm.
I am unaware of any feature that allows client to reserve cache pages to permanently store a set of predefined shared prefixes. Regular prefix caching will work for any shared prefix, but is subject to a few limitations that might limit how much benefit you see:
- Only full blocks that are a perfect match will be reused. For instance, if block size is 64 and your shared prefix has 180 tokens, only the first 128 tokens will be reused.
- Reusable KV cache blocks are evicted when the memory is needed for other purposes. On a system with high load, eviction is quite likely to occur, especially if some time passes between each use of the shared prefix.
from tensorrt-llm.
We can add a feature to reserve memory for shared prefix if client(s) request it. It would be added as one more input, probably named "prefix_length" that is an int32 value that says that the first "prefix_length" tokens of the request are part of a shared prefix. This would have to be an "advisory" field, so we would reserve space for the field if we can. Besides making the prefix more permanent by delaying eviction, we could remove the "full block" limitation so that the entire prefix could be shared, not just the part that fits into a whole number of blocks.
from tensorrt-llm.
@thorjohnsen thank you very much for your response. I think a prefix_length field, potentially with full block limitation removed, would be extremely useful. Essentially a way to indirectly sacrifice caching of blocks that are highly unlikely to be reused (e.g., end of the prompt) in order to have guaranteed caching of tokens that are known to be reused. I would like to assist with it but I don't think I have the time at the moment.
from tensorrt-llm.
This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 15 days."
from tensorrt-llm.
Related Issues (20)
- Different output with transformers lib and tensorrt llm when using lora HOT 2
- Latency increase when using Leader Mode in tritonserver HOT 2
- [Feature request] Mistral Large 2 support HOT 1
- [Bug] llama3.1-8b smoothquant error (use latest version: 5fa9436) HOT 1
- Wasteful computations in cross attention? HOT 1
- [Question] In-flight batching with Python
- branch dev-sm87-trt101 can works on Orin now ? HOT 6
- H20 infer Qwen14B fp8 results in a divide-by-zero error. HOT 1
- There are differences in the results of Qwen2-7B Instruction
- qwen1.5-7b---Why do I need 37GB of GPU memory HOT 1
- FP8Linear.forward cannot be called twice
- tag0.11: HOT 1
- Use 2 Lora in one request
- Segfault on main branch (problem in TopP layer) HOT 2
- Tensorrt llm support in Jetson Orion Nano HOT 1
- bugs in v0.10.0 version with tensorrtllm_backend HOT 1
- trtllm-build Mixtral-8x7B-v0.1 fp16 failed HOT 4
- [Feature Request] Executor API: Parametrized Logit Post Processors
- AttributeError: 'PluginConfig' object has no attribute '_remove_input_padding'. Did you mean: '_remove_input_padding'? HOT 1
- [Benchmark] Any scripts to benchmark throughput with a given dataset?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorrt-llm.