Comments (4)
symphonylyh
commented on August 17, 2024
1
@thanhlt998 releasing in this week's main branch release (today/tomorrow). Stay tuned.
from tensorrt-llm.
schetlur-nv
commented on August 17, 2024
@Oldpan for now we only support static batching for encoder-decoder models. But we plan inflight batching support in the coming weeks. Please keep an eye out. Feel free to reopen the issue if you have more questions.
from tensorrt-llm.
thanhlt998
commented on August 17, 2024
@schetlur-nv do you have any progresses here?
from tensorrt-llm.
Oldpan
commented on August 17, 2024
@symphonylyh
Thanks for the update, does this support encoders like nougat that require image inputs? I see that only bart and t5 are mentioned in the updated examples.
from tensorrt-llm.
Related Issues (20)
- Finding protobuf files while benchmarking TensorRT-LLM HOT 2
- How to add a new quantization method?
- How to preprocess a int4 model? HOT 1
- Does TensorRT-LLM support passing input_embeds directly？ HOT 11
- Convert_checkpoint.py failed with LLAMA 3.1 8B instruct HOT 2
- Will KOSMOS-2.5 be supported?
- Assertion failed: FMHA kernels are not found
- Why are the answers the same every time?
- Conver qwen2-57B-A14B failed
- `--use_fp8_context_fmha` is broken with Llama models
- Concerns about the removal of batch manager C++ API HOT 1
- Build failure when using FP8 quantized Medusa heads
- wgmma.mma_async instructions are serialized due to non wgmma instructions defining accumulator registers of a wgmma between start and end of the pipeline stage
- The weights compiled by v0.11.0 are run using TritonServer, and the concurrency is lower than that of the v0.10.0 version. The compilation scripts are the same.
- AttributeError: 'PluginConfig' object has no attribute '_streamingllm'. Did you mean: '_streamingllm'? HOT 4
- multi_block_mode has been removed in the current version? HOT 2
- TensorRT LLM does not stop while met the eos_id
- Llama3.1 support removed? HOT 2
- Cannot support long context input vs vllm.
- torch.OutOfMemoryError when try to build TensorRT engines for Qwen2-72B(-Instruct)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorrt-llm.