trtllm-build Mixtral-8x7B-v0.1 fp16 failed about tensorrt-llm

[Kefeng-Duan]

Hi, vonchenplus
Please decrease the max_batch_size/max_input_len/max_output_len when building the engine.
A rough estimation is

1. for Mixtral8x7b with fp16 precision, its weight consumes ~87GB
2. the fp16 KVcache consumes ~65536*2048(max_bs)*4096(max_in + max_out)*2(fp16)/1024/1024/1024 = 1024GB
   the sum of the two is far larger than your GPUs capacity (160GB)

[vonchenplus]

I have a few questions I'd like to ask:

1. When I set the workers parameter to 2 during trtllm-build, is only one GPU being used in the end? Because I see that only one GPU's memory is being utilized.
2. When you tested Mixtral-8x7B-v0.1 fp16, what graphics card did you use?

[Kefeng-Duan]

Hi, @vonchenplus

1. How do you observe that only one GPU was used? Could you help to double confirm your observation by print the workers here:

   TensorRT-LLM/tensorrt_llm/commands/build.py

   Line 408 in 93293aa

   workers = min(torch.cuda.device_count(), args.workers)

   ?
2. I think 2xA100 is OK

[vonchenplus]

Hi, @Kefeng-Duan ,

1. I checked yesterday and confirmed that the workers are set to 2, but it seems that only one card is being used.
2. However, in my current test, using two cards is indeed not working.

[Kefeng-Duan]

Hi, @vonchenplus

1. are your torch.cuda.device_count() = 1 or 2?
2. even when you decrease the max_bs/max_seqin/max_seqout to a proper range? what's the failure message?
   Thanks

[vonchenplus]

Hi @Kefeng-Duan

1. torch.cuda.device_count() is 2.
2. I didn't reduce max_bs/max_seqin/max_seqout because I wanted to reproduce the performance mentioned in your report.

[kaiyux]

Hi @vonchenplus , in the Mixtral-8x7B-v0.1.json file you shared, tp_size is set to 1, if you want to do TP2 with dummy weights, you'll need to set mapping as

"mapping": {
        "world_size": 2,
        "tp_size": 2,
        "pp_size": 1
    },

[vonchenplus]

Hi @kaiyux,

It's already working, thanks you.