Comments (6)
@sunnyqgg Could you please take a look? Thanks
from tensorrt-llm.
@tuanhe Which TRT version are you using? This is for TRT 10.1
from tensorrt-llm.
@tuanhe Which TRT version are you using? This is for TRT 10.1
Sure, I had updated Tensorrt to 10.1
from tensorrt-llm.
@sunnyqgg Could you kindly share me the steps to follow your work?
from tensorrt-llm.
@sunnyqgg Hi there , any updates?
from tensorrt-llm.
HI @tuanhe , Actually this branch is for other uses, and we only did few verifications.
If you want to use:
step1: install trt 10.1 and cuda 12.x
step2: python3 scripts/build_wheel.py --cuda_architectures "87-real" --trt_root your_trt_path
from tensorrt-llm.
Related Issues (20)
- Finding protobuf files while benchmarking TensorRT-LLM HOT 2
- How to add a new quantization method?
- How to preprocess a int4 model? HOT 1
- Does TensorRT-LLM support passing input_embeds directly? HOT 11
- Convert_checkpoint.py failed with LLAMA 3.1 8B instruct HOT 2
- Will KOSMOS-2.5 be supported?
- Assertion failed: FMHA kernels are not found
- Why are the answers the same every time?
- Conver qwen2-57B-A14B failed
- `--use_fp8_context_fmha` is broken with Llama models
- Concerns about the removal of batch manager C++ API HOT 1
- Build failure when using FP8 quantized Medusa heads
- wgmma.mma_async instructions are serialized due to non wgmma instructions defining accumulator registers of a wgmma between start and end of the pipeline stage
- The weights compiled by v0.11.0 are run using TritonServer, and the concurrency is lower than that of the v0.10.0 version. The compilation scripts are the same.
- AttributeError: 'PluginConfig' object has no attribute '_streamingllm'. Did you mean: '_streamingllm'? HOT 4
- multi_block_mode has been removed in the current version? HOT 2
- TensorRT LLM does not stop while met the eos_id
- Llama3.1 support removed? HOT 2
- Cannot support long context input vs vllm.
- torch.OutOfMemoryError when try to build TensorRT engines for Qwen2-72B(-Instruct)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorrt-llm.