premai-io / benchmarks Goto Github PK

Fix the supported features table, to better reflect reality.
Cleanup the code and ensure names for user exposed ENV variables are consistent.
Handle quantization information, and provide info regarding quantization method used across models and scripts in the README. (llama.cpp quantization vs tinygrad vs CTranslate2 vs GPTQ)
Provide more CLI options for running individual scripts with other models as well. For testing frameworks on systems with less memory.

Feature(in order of priority):

Add custom model runner code, for running benchmarks and provide hooks for directly reporting performance metrics into as an output.
Setup scripts for running benchmarks with a single command and getting proper performance reports.
Improve caching for models, currently some scripts will end up redownloading the models, which has already been fixed for some.
Simplify running for any specific platform(nvidia/mac), with any supported model.
Auto-setup rust, python, git and etc, for the user before running the benchmark(low priority).

Enhance Inference Engines - Additional Engines

inference

see here

Kernl

see here

OpenLLM

see here

voltaML

see here

substratus

see here

Petals

Test with a private swarm (refer to premAI-io/dev-portal#69)

Questions

Where are the bottlenecks?
There are no advantages to build_gpu? there's no way to force using both the local GPU and the swarm (without connecting the local GPU to the swarm)?
Compare Petals with Deepspeed in a centralized scenario.

candle-vllm

see here

PyTorch

PyTorch (Transformers) (test multiple versions eg 1.2.1 vs 2.1.0)

ctransformers

see here

Linter

Description

Most of the repo is in Python. Set up a linter accordingly. Check prem-daemon accordingly.

Truss

see here

Ollama

see here

text generation webui

see here

AWQ Quantization

Description

Benchmarks for inference engines supporting this quantization method.

Medusa

see here

Enhance Inference Engines - Core

TensorRT LLM

see here

tvm

see here

Leapfrog AI

see here

agenta

see here

Text Generation Inference

see here

Two commands to run all the benchmarks

Description

The repo should expose two commands:

Command to run benchmarks on Mac (CPU/GPU when available)
Command to run benchmarks on NVIDIA GPUs

I should be able to clone the repo git clone and run bash ./mac.sh or bash ./nvidia.sh. If you want you can have multiple abstractions and CLI exposed, but this is the final objective.

Standard output should print the results in a consistent manner in order to be able to check them easily.