Comments (9)
Could you share the end to end reproduced steps?
Also, how to make GPU initial usage is 100%? Running another program at the same time?
from tensorrt-llm.
Hi,
We had kind of similar issue, for looping and sending many times the same input data to stress test the GPU.
It got us some NaN after hundred of thousands of inference (again same data all the time, and synchronizing everywhere it made sense)! Model follows Roberta architecture.
We noticed that another Roberta based architecture, compiled the same way had not such issue, so we thought it s weight related. We switched to BF16, keeping the fp16 optimized kernel for flash attention, during build trt complains a lot that some nodes gets bf16 and fp16 inputs (I guess to let us know it adds some casting to many places), still e2e perf are almost the same, output precision is slightly lower (compared to fp32 AMP reference model), but no more issue, and we stress tested it quite a lot since then.
May be something to test @0xd8b ?
from tensorrt-llm.
We converted the T5 model using the files in example/enc_dec/. The data type used for conversion is float16 (batch_size=1, strongly_typed=True, use_bert_plugin=True). Additionally, we truncated the output of hidden_states. However, during high GPU utilization, the encoder output becomes NaN.
Yes, we concurrently ran another model inference program to achieve 100% GPU utilization, yet only 1/10 of the total memory was utilized.
An intriguing observation is that NaN occurrences do not happen during TRT conversion with float32 data types. Additionally, in float16 types, if the GPU's initial utilization is 0%, inference proceeds normally. From this observation, it seems that the issue is not solely related to data overflow.
from tensorrt-llm.
@pommedeterresautee thans for your reply! Are you referring to the conversion of the model using the bfloat16 data type
from tensorrt-llm.
Yes, The conversion to bf16 is done during conversion step. We had to modify the part where it binds weights to trt engine. Have a check to _utils of trt lib, there are some useful stuff for bf16.
from tensorrt-llm.
Fwiw back in the time I wrote a bunch of custom triton (language not server) kernels and T5 weights were super hard to manage in fp16, largest flavors NaN from time to time depending of the input. It's a long time i didn't touch it, but from what I barely remember Google trained it in bf16. I know at first view it's not related to gpu occupation but may be something to keep in mind.
from tensorrt-llm.
@pommedeterresautee Ok, thanks for the suggestion, I will give it a try. However, I'm still curious as to why the model (float16 type) works fine at low gpu usage.
from tensorrt-llm.
We attempted to convert the model to bfloat16 and conduct inference, yet the issue persists even under high GPU utilization. It seems that there's a problem occurring in the computation of the RMSnorm layer."
from tensorrt-llm.
The issue is also caused by the encoder_input_length problem described in #1847. This issue can be closed.
from tensorrt-llm.
Related Issues (20)
- return-generation-logits bug when fp8 enabled
- Segmentation fault when building engine
- questions on return-generation-logits when streaming
- Fail to quantize qwen2-72b with FP8 HOT 1
- [Question] What is the difference between gptManagerBenchmark and gptSessionBenchmark in cpp benchmarks?
- [0.11.0] Model Conversion OOMed in P4D.24xl
- Random generation when inferring llama 2 with and without LoRA in the same batch
- Add support for Nougat model
- Error trying to build the visual encoder for llava-v1.6-34b-hf using build_visual_engine.py HOT 3
- Finding protobuf files while benchmarking TensorRT-LLM HOT 2
- How to add a new quantization method?
- How to preprocess a int4 model? HOT 1
- Does TensorRT-LLM support passing input_embeds directly? HOT 11
- Convert_checkpoint.py failed with LLAMA 3.1 8B instruct HOT 2
- Will KOSMOS-2.5 be supported?
- Assertion failed: FMHA kernels are not found
- Why are the answers the same every time?
- Conver qwen2-57B-A14B failed
- `--use_fp8_context_fmha` is broken with Llama models
- Concerns about the removal of batch manager C++ API HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorrt-llm.