Comments (8)
Hi, vonchenplus
Please decrease the max_batch_size/max_input_len/max_output_len when building the engine.
A rough estimation is
- for Mixtral8x7b with fp16 precision, its weight consumes ~87GB
- the fp16 KVcache consumes ~65536*2048(max_bs)*4096(max_in + max_out)*2(fp16)/1024/1024/1024 = 1024GB
the sum of the two is far larger than your GPUs capacity (160GB)
from tensorrt-llm.
I have a few questions I'd like to ask:
- When I set the workers parameter to 2 during trtllm-build, is only one GPU being used in the end? Because I see that only one GPU's memory is being utilized.
- When you tested Mixtral-8x7B-v0.1 fp16, what graphics card did you use?
from tensorrt-llm.
Hi, @vonchenplus
- How do you observe that only one GPU was used? Could you help to double confirm your observation by print the workers here:
TensorRT-LLM/tensorrt_llm/commands/build.py
Line 408 in 93293aa
- I think 2xA100 is OK
from tensorrt-llm.
Hi, @Kefeng-Duan ,
- I checked yesterday and confirmed that the workers are set to 2, but it seems that only one card is being used.
- However, in my current test, using two cards is indeed not working.
from tensorrt-llm.
Hi, @vonchenplus
- are your torch.cuda.device_count() = 1 or 2?
- even when you decrease the max_bs/max_seqin/max_seqout to a proper range? what's the failure message?
Thanks
from tensorrt-llm.
Hi @Kefeng-Duan
- torch.cuda.device_count() is 2.
- I didn't reduce max_bs/max_seqin/max_seqout because I wanted to reproduce the performance mentioned in your report.
from tensorrt-llm.
Hi @vonchenplus , in the Mixtral-8x7B-v0.1.json file you shared, tp_size
is set to 1, if you want to do TP2 with dummy weights, you'll need to set mapping as
"mapping": {
"world_size": 2,
"tp_size": 2,
"pp_size": 1
},
from tensorrt-llm.
Hi @kaiyux,
It's already working, thanks you.
from tensorrt-llm.
Related Issues (20)
- Finding protobuf files while benchmarking TensorRT-LLM HOT 2
- How to add a new quantization method?
- How to preprocess a int4 model? HOT 1
- Does TensorRT-LLM support passing input_embeds directly? HOT 11
- Convert_checkpoint.py failed with LLAMA 3.1 8B instruct HOT 2
- Will KOSMOS-2.5 be supported?
- Assertion failed: FMHA kernels are not found
- Why are the answers the same every time?
- Conver qwen2-57B-A14B failed
- `--use_fp8_context_fmha` is broken with Llama models
- Concerns about the removal of batch manager C++ API HOT 1
- Build failure when using FP8 quantized Medusa heads
- wgmma.mma_async instructions are serialized due to non wgmma instructions defining accumulator registers of a wgmma between start and end of the pipeline stage
- The weights compiled by v0.11.0 are run using TritonServer, and the concurrency is lower than that of the v0.10.0 version. The compilation scripts are the same.
- AttributeError: 'PluginConfig' object has no attribute '_streamingllm'. Did you mean: '_streamingllm'? HOT 4
- multi_block_mode has been removed in the current version? HOT 2
- TensorRT LLM does not stop while met the eos_id
- Llama3.1 support removed? HOT 2
- Cannot support long context input vs vllm.
- torch.OutOfMemoryError when try to build TensorRT engines for Qwen2-72B(-Instruct)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorrt-llm.