premai-io / benchmarks

🕹️ Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models.

🕹️ Benchmarks

A fully reproducible Performance Comparison of MLOps Engines, Frameworks, and Languages on Mainstream AI Models

Check out our release blog to know more.

Table of Contents

1. Quick glance towards performance metrics
2. ML Engines
3. Why Benchmarks
4. Usage and workflow
5. Contribute

🥽 Quick glance towards performance benchmarks

Take a first glance at Mistral 7B v0.1 Instruct and Llama 2 7B Chat Performance Metrics Across Different Precision and Inference Engines. Here is our run specification that generated this performance benchmark reports.

Environment:

• Model: Mistral 7B v0.1 Instruct / Llama 2 7B Chat
• CUDA Version: 12.1
• Batch size: 1

Command:

./benchmark.sh --repetitions 10 --max_tokens 512 --device cuda --model mistral/llama --prompt 'Write an essay about the transformer model architecture'

Mistral 7B v0.1 Instruct

Performance Metrics: (unit: Tokens/second)

Engine	float32	float16	int8	int4
transformers (pytorch)	39.61 ± 0.65	37.05 ± 0.49	5.08 ± 0.01	19.58 ± 0.38
AutoAWQ	-	-	-	63.12 ± 2.19
AutoGPTQ	39.11 ± 0.42	42.94 ± 0.80
DeepSpeed		79.88 ± 0.32
ctransformers	-	-	86.14 ± 1.40	87.22 ± 1.54
llama.cpp	-	-	88.27 ± 0.72	95.33 ± 5.54
ctranslate	43.17 ± 2.97	68.03 ± 0.27	45.14 ± 0.24	-
PyTorch Lightning	32.79 ± 2.74	43.01 ± 2.90	7.75 ± 0.12	-
Nvidia TensorRT-LLM	117.04 ± 2.16	206.59 ± 6.93	390.49 ± 4.86	427.40 ± 4.84
vllm	84.91 ± 0.27	84.89 ± 0.28	-	106.03 ± 0.53
exllamav2	-	-	114.81 ± 1.47	126.29 ± 3.05
onnx	15.75 ± 0.15	22.39 ± 0.14	-	-
Optimum Nvidia	50.77 ± 0.85	50.91 ± 0.19	-	-

Performance Metrics: GPU Memory Consumption (unit: MB)

Engine	float32	float16	int8	int4
transformers (pytorch)	31071.4	15976.1	10963.91	5681.18
AutoGPTQ	13400.80	6633.29
AutoAWQ	-	-	-	6572.47
DeepSpeed		80097.34
ctransformers	-	-	10255.07	6966.74
llama.cpp	-	-	9141.49	5880.41
ctranslate	32602.32	17523.8	10074.72	-
PyTorch Lightning	48783.95	18738.05	10680.32	-
Nvidia TensorRT-LLM	79536.59	78341.21	77689.0	77311.51
vllm	73568.09	73790.39	-	74016.88
exllamav2	-	-	21483.23	9460.25
onnx	33629.93	19537.07	-	-
Optimum Nvidia	79563.85	79496.74	-	-

*(Data updated: 30th April 2024)

Llama 2 7B Chat

Performance Metrics: (unit: Tokens / second)

Engine	float32	float16	int8	int4
transformers (pytorch)	36.65 ± 0.61	34.20 ± 0.51	6.91 ± 0.14	17.83 ± 0.40
AutoAWQ	-	-	-	63.59 ± 1.86
AutoGPTQ	34.36 ± 0.51	36.63 ± 0.61
DeepSpeed		84.60 ± 0.25
ctransformers	-	-	85.50 ± 1.00	86.66 ± 1.06
llama.cpp	-	-	89.90 ± 2.26	97.35 ± 4.71
ctranslate	46.26 ± 1.59	79.41 ± 0.37	48.20 ± 0.14	-
PyTorch Lightning	38.01 ± 0.09	48.09 ± 1.12	10.68 ± 0.43	-
Nvidia TensorRT-LLM	104.07 ± 1.61	191.00 ± 4.60	316.77 ± 2.14	358.49 ± 2.38
vllm	89.40 ± 0.22	89.43 ± 0.19	-	115.52 ± 0.49
exllamav2	-	-	125.58 ± 1.23	159.68 ± 1.85
onnx	14.28 ± 0.12	19.42 ± 0.08	-	-
Optimum Nvidia	53.64 ± 0.78	53.82 ± 0.11	-	-

Performance Metrics: GPU Memory Consumption (unit: MB)

Engine	float32	float16	int8	int4
transformers (pytorch)	29114.76	14931.72	8596.23	5643.44
AutoAWQ	-	-	-	7149.19
AutoGPTQ	10718.54	5706.35
DeepSpeed		80105.13
ctransformers	-	-	9774.83	6889.14
llama.cpp	-	-	8797.55	5783.95
ctranslate	29951.52	16282.29	9470.74	-
PyTorch Lightning	42748.35	14736.69	8028.16	-
Nvidia TensorRT-LLM	79421.24	78295.07	77642.86	77256.98
vllm	77928.07	77928.07	-	77768.69
exllamav2	-	-	16582.18	7201.62
onnx	33072.09	19180.55	-	-
Optimum Nvidia	79429.63	79295.41	-	-

*(Data updated: 30th April 2024)

Our latest version benchmarks Llama 2 7B chat and Mistral 7B v0.1 instruct. The latest version only benchmarks on A100 80 GPU. Because our primary focus is enterprises. Our previous versions benchmarked Llama 2 7B on Cuda and Mac (M1/M2) CPU and metal. You can find those in the archive.md file. Please note that those numbers are old because all the engines are maintained properly continuously with improvements. So those numbers might be a bit outdated.

🛳 ML Engines

In the current market, there are several ML Engines. Here is a quick glance at all the engines used for the benchmark and a quick summary of their support matrix. You can find the details about the nuances here.

Engine	Float32	Float16	Int8	Int4	CUDA	ROCM	Mac M1/M2	Training
candle	⚠️	✅	⚠️	⚠️	✅	❌	🚧	❌
llama.cpp	❌	❌	✅	✅	✅	🚧	🚧	❌
ctranslate	✅	✅	✅	❌	✅	❌	🚧	❌
onnx	✅	✅	❌	❌	✅	⚠️	❌	❌
transformers (pytorch)	✅	✅	✅	✅	✅	🚧	✅	✅
vllm	✅	✅	❌	✅	✅	🚧	❌	❌
exllamav2	❌	❌	✅	✅	✅	🚧	❌	❌
ctransformers	❌	❌	✅	✅	✅	🚧	🚧	❌
AutoGPTQ	✅	✅	⚠️	⚠️	✅	❌	❌	❌
AutoAWQ	❌	❌	❌	✅	✅	❌	❌	❌
DeepSpeed-MII	❌	✅	❌	❌	✅	❌	❌	⚠️
PyTorch Lightning	✅	✅	✅	✅	✅	⚠️	⚠️	✅
Optimum Nvidia	✅	✅	❌	❌	✅	❌	❌	❌
Nvidia TensorRT-LLM	✅	✅	✅	✅	✅	❌	❌	❌

Legend:

• ✅ Supported
• ❌ Not Supported
• ⚠️ There is a catch related to this
• 🚧 It is supported but not implemented in this current version

You can check out the nuances related to ⚠️ and 🚧 in details here

🤔 Why Benchmarks

This can be a common question. What are the benefits you can expect from this repository? So here are some quick pointers to answer those.

1. Oftentimes, we are confused when given several choices on which engines or precision to use for our LLM inference workflow. Because sometimes we have constraints on computing and sometimes we have other requirements. So this repository helps you to get a quick idea of what to use based on your requirements.

2. Sometimes there comes a quality vs speed tradeoff between engines and precisions. So this repository keeps track of those and gives you an idea to understand the tradeoffs so that you can give more importance to your priorities.

3. A fully reproducible and hackable script. The latest benchmarks come with a lot of best practices so that they can be robust enough to run on GPU devices. Also, you can reference and extend the implementations to build your own workflows out of it.

🚀 Usage and workflow

Welcome to our benchmarking repository! This organized structure is designed to simplify benchmark management and execution. Each benchmark runs an inference engine that provides some sort of optimizations either through just quantization or device-specific optimizations like custom cuda kernels.

To get started you need to download the models first. This will download the following models: Llama2 7B Chat and Mistral-7B v0.1 Instruct. You can start download by typing this command:

./download.sh

Please make sure that when you are running Llama2-7B Chat weights, we would assume that you already agreed to the required terms and conditions and got verified to download the weights.

A Benchmark workflow

When you run a benchmark, the following set of events occurs:

• Automatically setting up the environments and installing the required dependencies.

• Converting the models to some specific format (if required) and saving them.

• Running the benchmarks and storing them inside the logs folder. Each log folder has the following structure:

• performance.log: This will track the model run performances. You can see the token/sec and memory consumption (MB) here.

• quality.md: This file is an automatically generated readme file, which contains qualitative comparisons of different precisions of some engines. We take 5 prompts and run them for the set of supported precisions of that engine. We then put those results side by side. Our ground truth is the output from huggingface PyTorch model with raw float32 weights.

• quality.json Same as the readme file but more in raw format.

Inside each benchmark folder, you will also see a readme.md file which contains all the information and the qualitative comparison of the engine. For example: bench_tensorrtllm.

Running a Benchmark

Here is how we run benchmarks for an inference engine.

./bench_<engine-name>/bench.sh \
 --prompt <value> \ # Enter a prompt string
 --max_tokens <value> \  # Maximum number of tokens to output
 --repetitions <value> \  # Number of repetitions to be made for the prompt.
 --device <cpu/cuda/metal> \  # The device in which we want to benchmark.
 --model_name <name-of-the-model> # The name of the model. (options: 'llama' for Llama2 and 'mistral' for Mistral-7B-v0.1)

Here is an example. Let's say we want to benchmark Nvidia TensorRT LLM. So here is how the command would look like:

./bench_tensorrtllm/bench.sh -d cuda -n llama -r 10

To know more, here is more detailed info on each command line argument.

 -p, --prompt Prompt for benchmarks (default: 'Write an essay about the transformer model architecture')
 -r, --repetitions Number of repetitions for benchmarks (default: 10)
 -m, --max_tokens Maximum number of tokens for benchmarks (default: 512)
 -d, --device Device for benchmarks (possible values: 'metal', 'cuda', and 'CPU', default: 'cuda')
 -n, --model_name The name of the model to benchmark (possible values: 'llama' for using Llama2, 'mistral' for using Mistral 7B v0.1)
 -lf, --log_file Logging file name.
 -h, --help Show this help message

🤝 Contribute

We welcome contributions to enhance and expand our benchmarking repository. If you'd like to contribute a new benchmark, follow these steps:

Creating a New Benchmark

1. Create a New Folder

Start by creating a new folder for your benchmark. Name it bench_{new_bench_name} for consistency.

mkdir bench_{new_bench_name}

2. Folder Structure

Inside the new benchmark folder, include the following structure

bench_{new_bench_name}
├── bench.sh # Benchmark script for setup and execution
├── requirements.txt # Dependencies required for the benchmark
└── ... # Any additional files needed for the benchmark

3. Benchmark Script (bench.sh):

The bench.sh script should handle setup, environment configuration, and the actual execution of the benchmark. Ensure it supports the parameters mentioned in the Benchmark Script Parameters section.

Pre-commit Hooks

We use pre-commit hooks to maintain code quality and consistency.

1. Install Pre-commit: Ensure you have pre-commit installed

pip install pre-commit

2. Install Hooks: Run the following command to install the pre-commit hooks

pre-commit install

The existing pre-commit configuration will be used for automatic checks before each commit, ensuring code quality and adherence to defined standards.

benchmarks's Issues

candle-vllm

see here

Kernl

see here

stable-fast

see here

Inference Servers

• Ray Serve
• OpenLLM
• Triton Inference Server

Enhance Inference Acceleration Toolkits

• AITemplate
• VoltaML

Torch Serve

see here

Roadmap - Task List

Minor fixes:

• Fix the supported features table, to better reflect reality.
• Cleanup the code and ensure names for user exposed ENV variables are consistent.
• Handle quantization information, and provide info regarding quantization method used across models and scripts in the README. (llama.cpp quantization vs tinygrad vs CTranslate2 vs GPTQ)
• Provide more CLI options for running individual scripts with other models as well. For testing frameworks on systems with less memory.

Feature(in order of priority):

• Add custom model runner code, for running benchmarks and provide hooks for directly reporting performance metrics into as an output.
• Setup scripts for running benchmarks with a single command and getting proper performance reports.
• Improve caching for models, currently some scripts will end up redownloading the models, which has already been fixed for some.
• Simplify running for any specific platform(nvidia/mac), with any supported model.
• Auto-setup rust, python, git and etc, for the user before running the benchmark(low priority).

WasmEdge Inference

Description

• https://www.secondstate.io/articles/wasm-runtime-agi/

inference

see here

RayLLM

see here

MLC LLM

see here

webonnx

see here

vllm

see here

AITemplate

see here

text generation webui

see here

Linter

Description

Most of the repo is in Python. Set up a linter accordingly. Check prem-daemon accordingly.

Enhance Inference Servers

• Text Generation Inference
• TorchServe
• Willow Inference Server

Enhance Inference Engines - Core

• ONNX Runtime
• TensorRT
• TensorRT-LLM

Medusa

see here

RayLLM

see here

Text Embeddings Inference

see here

Ray Serve

see here

Leapfrog AI

see here

AWQ Quantization

Description

Benchmarks for inference engines supporting this quantization method.

Ollama

see here

Improve repo to go Public

Description

• Cool README with CTAs to the products.
• Tags, topics
• Description

Inference Acceleration Toolkits

• MLC LLM
• tvm
• vLLM

Enhance Inference Engines - Additional Engines

• PyTorch (Transformers) (test multiple versions eg 1.2.1 vs 2.1.0)
• ctransformers
• wonnx (wgpu + ONNX)
• stable-fast
• candle-vllm

PyTorch

PyTorch (Transformers) (test multiple versions eg 1.2.1 vs 2.1.0)

Truss

see here

Willow Inference Server

see here

fastembed

see here

Text Generation Inference

see here

agenta

see here

TensorRT

see here

Petals

Test with a private swarm (refer to premAI-io/dev-portal#69)

Questions

• Where are the bottlenecks?
• There are no advantages to build_gpu? there's no way to force using both the local GPU and the swarm (without connecting the local GPU to the swarm)?
• Compare Petals with Deepspeed in a centralized scenario.

OpenLLM

see here

tvm

see here

DeepSpeed

see here

voltaML

see here

RunPod

see here

OneDiffusion

see here

Burn Upgrade

Upgrade burn version from 0.9.0 to 0.10.0 in llama burn

nuclio

see here

TensorRT LLM

see here

substratus

see here

ONNX Runtime

see here

Two commands to run all the benchmarks

Description

The repo should expose two commands:

• Command to run benchmarks on Mac (CPU/GPU when available)
• Command to run benchmarks on NVIDIA GPUs

I should be able to clone the repo git clone and run bash ./mac.sh or bash ./nvidia.sh. If you want you can have multiple abstractions and CLI exposed, but this is the final objective.

Standard output should print the results in a consistent manner in order to be able to check them easily.

Triton Inference Server

see here

ctransformers

see here