elvischenv/sglang
SGLang is a fast serving framework for large language models and vision language models.
Repositories
elvischenv/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
elvischenv/flashinfer
FlashInfer: Kernel Library for LLM Serving
elvischenv/InferenceX
Open Source Continuous Inference Benchmark Research Platform — Kimi K2.7-Code, MiniMax M3, DeepSeekv4, GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72 & soon™ TPUv6e/v7/Trainium2/3 | 开源持续推理基准研究平台 — Kimi K2.7-Code、MiniMax M3、DeepSeekv4、GLM5 - GB200 NVL72 vs MI355X vs B200 vs GB300 NVL72，即将推出™ TPUv6e/v7/Trainium2/3
elvischenv/srt-slurm
NVIDIA Inference Benchmarks provide recipes in ready-to-use templates for evaluating platform speed. Validate your platform across specific AI use cases across hardware and software combinations.
elvischenv/TensorRT-LLM
TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and support state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in performant way.
elvischenv/lm-evaluation-harness
A framework for few-shot evaluation of language models.
elvischenv/srt-slurm-legacy
Benchmark SGLang on SLURM