Comments (5)
Thanks @lopuhin, will update you very soon
from benchmarks.
Hey, sorry for the late reply, however, upon seeing that, I did cross check, and it is to be noted that the device I am using is an A100 80 GB and also I maxed out the gpu utilization (with higher kv cache), so that is why there is so much high throughput.
Speaking of the code, I also printed out the line, and it seems fine, otherwise, the output that you see here would have been very different from ground truth.
from benchmarks.
Hey, thanks for looking into this and no worries. And btw thanks for a really nicely reproducible code, it was super helpful, as trt-llm is showing really good scaling with larger batch sizes for us.
I think there is a way to see that there is something strange going on, without any code changes -- to observe how reported generation speed changes with different prompts, which have different generation lengths.
Using only int4 models for simplicity, L4 GPU:
$ ./bench_tensorrtllm/bench.sh -d cuda -n mistral -r 5
(repo) 2024-06-03 10:42:20,290 - INFO - mistral-Nvidia-TRT-LLM (token/sec), int4: 248.57 ± 8.80
(fixed) 2024-06-03 10:37:06,271 - INFO - mistral-Nvidia-TRT-LLM (token/sec), int4: 59.04 ± 2.87
$ ./bench_tensorrtllm/bench.sh -d cuda -n mistral -r 5 -p "just say hi and nothing else, ok?"
(repo) 2024-06-03 10:43:02,444 - INFO - mistral-Nvidia-TRT-LLM (token/sec), int4: 7617.51 ± 2783.96
(fixed) 2024-06-03 10:39:22,647 - INFO - mistral-Nvidia-TRT-LLM (token/sec), int4: 45.19 ± 16.50
$ ./bench_tensorrtllm/bench.sh -d cuda -n mistral -r 5 -p "write a super long essey about transformers, never stop"
(repo) 2024-06-03 10:44:19,333 - INFO - mistral-Nvidia-TRT-LLM (token/sec), int4: 176.23 ± 4.39
(fixed) 2024-06-03 10:40:57,054 - INFO - mistral-Nvidia-TRT-LLM (token/sec), int4: 59.70 ± 1.50
I expect that A100 would have better absolute performance, but similar relative performance -- when generation is short, the token/s as measured by current code is off the charts (7600 t/s), and when it's long, it's much slower (176 t/s). While with the fix the generation speed is slightly lower when generation is short (due to kernel launch overhead), and consistent with other measurements. This is happening because the measured "num_output_tokens": len(output_tokens),
is around 1.5k for all generations, long or short.
Curious if you see similar discrepancy in reported generation speed with TensoRT-LLM with different prompts? Could be an issue with my build, sorry if that's the case.
otherwise, the output that you see here would have been very different from ground truth.
I think extra EOS tokens in the end would be ignored during decoding, so the output is still fine, it's only how we measure the number of generated tokens is off.
from benchmarks.
I expect that A100 would have better absolute performance, but similar relative performance -- when generation is short, the token/s as measured by current code is off the charts (7600 t/s), and when it's long, it's much slower (176 t/s). While
Wow, 7K tokens/sec, this is concerning, well I saw some discrepencies where the tokens/sec was differing by 100 (like sometimes getting 400 and sometimes 300), but one way to analyse this, is out of 10 generations, I am going to print all of them and see if all of them outputs the same thing or not.
Also, I see you using two envs, can you show me what you changed in the fixed
env?
Thanks
from benchmarks.
Makes sense, thank you! If you can print the output_tokens
variable inside run_model
in bench.py
that should show the issue I think.
Also, I see you using two envs, can you show me what you changed in the fixed env?
Sure, here is a mistral-specific fix:
$ git diff bench_tensorrtllm/bench.py
diff --git a/bench_tensorrtllm/bench.py b/bench_tensorrtllm/bench.py
index 22fb1e6..b44be44 100644
--- a/bench_tensorrtllm/bench.py
+++ b/bench_tensorrtllm/bench.py
@@ -100,9 +100,15 @@ class TensorRTLLMBenchmark(BaseBenchmarkClass):
output_ids = output["output_ids"]
output_tokens = output_ids[0][0].detach().cpu().tolist()[num_input_tokens:]
+ eos = 2 # FIXME this is mistral-only
+ if eos in output_tokens:
+ num_output_tokens = output_tokens.index(eos) + 1
+ else:
+ num_output_tokens = len(output_tokens)
+
return {
"output_tokens": output_tokens,
- "num_output_tokens": len(output_tokens),
+ "num_output_tokens": num_output_tokens,
}
from benchmarks.
Related Issues (20)
- A template file for only tables as a single source of truth.
- Latest benchmarks not updating.
- Update and re-iterate the missing parts of benchmarks.
- Additional command line argument `--clean` for clean installation.
- A nice blog containing all the analysis of benchmarks HOT 1
- LM Deploy
- ML Engines
- Complete ML Engines Table HOT 1
- New engines
- Improvements on logging and storing results HOT 1
- For quality checks reference should be taken from actual PyTorch version of model HOT 1
- Comments on output quality on main README HOT 2
- Change Main readme HOT 1
- Setup new md files from templates HOT 1
- Add `torch.inference_session` on runner function
- Add model loading time for each benchmarks
- Use exllamav2's smart 4-bit KV cache for memory benchmark HOT 1
- aphrodite-engine
- Max Engine by Mojo
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from benchmarks.