njhill/flashinfer
FlashInfer: Kernel Library for LLM Serving
Repositories
njhill/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
njhill/ci-infra
This repo hosts code for vLLM CI & Performance Benchmark infrastructure.
njhill/kv-utils
Abstracted helper classes providing consistent key-value store functionality, with zookeeper and etcd3 implementations
njhill/etcd-java
Alternative etcd3 java client
njhill/srt-slurm
NVIDIA Inference Benchmarks provide recipes in ready-to-use templates for evaluating platform speed. Validate your platform across specific AI use cases across hardware and software combinations.
njhill/fastsafetensors
High-performance safetensors model loader
njhill/vllm-tgis-adapter
vLLM adapter for a TGIS-compatible gRPC server.
njhill/LMCache
Redis for LLMs
njhill/text-generation-inference
Large Language Model Text Generation Inference
njhill/sdg
Python library for Synthetic Data Generation
njhill/flash-attention
Fast and memory-efficient exact attention
njhill/optimum
🏎️ Accelerate training and inference of 🤗 Transformers with easy to use hardware optimization tools
njhill/transformers
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.
njhill/transformers-bloom-inference
Fast Inference Solutions for BLOOM
njhill/manifestival
Create, manipulate, apply and delete Kubernetes resource manifests at runtime
njhill/MLServer
An inference server for your machine learning models, including support for multiple frameworks, multi-model serving and more
njhill/modelmesh
Distributed Model Serving Framework
njhill/ibm-common-service-operator
Operator to install IBM Common Services
njhill/modelmesh-runtime-adapter
Unified runtime-adapter image of the sidecar containers which run in the modelmesh pods
njhill/rest-proxy
KServe V2 Protocol Rest API Implementation Proxy
njhill/kfserving
Serverless Inferencing on Kubernetes
njhill/litelinks-core
High-performance netty and thrift-based microservice RPC library for Java
njhill/etcd
Distributed reliable key-value store for the most critical data of a distributed system
njhill/concurrentlinkedhashmap
A ConcurrentLinkedHashMap for Java
njhill/serving
A flexible, high-performance serving system for machine learning models
njhill/argo
Argo Workflows: Get stuff done with Kubernetes.
njhill/metacontroller
Lightweight Kubernetes controllers as a service
njhill/JCTools
njhill/grpc
The C based gRPC (C++, Python, Ruby, Objective-C, PHP, C#)