jdebache/flashinfer
FlashInfer: Kernel Library for LLM Serving
Repositories
jdebache/TensorRT-Model-Optimizer
nvidia-modelopt is a unified library of state-of-the-art model optimization techniques like quantization, pruning, distillation, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM or TensorRT to optimize inference speed.
jdebache/DeepEP
DeepEP: an efficient expert-parallel communication library
jdebache/nixl
NVIDIA Inference Xfer Library (NIXL)
jdebache/dynamo
A Datacenter Scale Distributed Inference Serving Framework
jdebache/TensorRT-LLM
TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.
jdebache/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
jdebache/recipes
Common recipes to run vLLM
jdebache/tokenizers
💥 Fast State-of-the-Art Tokenizers optimized for Research and Production
jdebache/transformers
🤗 Transformers: the model-definition framework for state-of-the-art machine learning models in text, vision, audio, and multimodal models, for both inference and training.
jdebache/lm-evaluation-harness
A framework for few-shot evaluation of language models.
jdebache/numba
NumPy aware dynamic Python compiler using LLVM
jdebache/llvmlite
A lightweight LLVM python binding for writing JIT compilers
jdebache/FlashMLA
FlashMLA: Efficient MLA kernels
jdebache/cutlass
CUDA Templates for Linear Algebra Subroutines
jdebache/pytorch
Tensors and Dynamic neural networks in Python with strong GPU acceleration
jdebache/buck2
Build system, successor to Buck
jdebache/spdlog
Fast C++ logging library.
jdebache/mscclpp
MSCCL++: A GPU-driven communication stack for scalable AI applications
jdebache/dlpack
common in-memory tensor structure
jdebache/gallery
jdebache/nvbench
CUDA Kernel Benchmarking Library
jdebache/rules_cuda
Starlark implementation of bazel rules for CUDA.
jdebache/bazel-compile-commands-extractor
Goal: Enable awesome tooling for Bazel users of the C language family.
jdebache/vscode-bazel
Bazel support for Visual Studio Code
jdebache/rules_cc
C++ Rules for Bazel
jdebache/tensorflow
An Open Source Machine Learning Framework for Everyone
jdebache/TensorRT
NVIDIA® TensorRT™, an SDK for high-performance deep learning inference, includes a deep learning inference optimizer and runtime that delivers low latency and high throughput for inference applications.