hhzhang16/llm-d-async
Asynchronous Processor for Inference Gateway. Orchestrator of queues
Repositories
hhzhang16/llm-d-batch-gateway
llm-d-batch-gateway is a standalone, backend-agnostic, OpenAI-compatible batch API and processing engine
hhzhang16/frontend-crates
hhzhang16/TensorRT-LLM
TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT LLM also contains components to create Python and C++ runtimes that orchestrate the inference execution in a performant way.
hhzhang16/LibriStutterData
LibriStutter data for a class final project. Will remove after the project ends!
hhzhang16/triton_distributed
hhzhang16/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
hhzhang16/CS248-Final-Project
hhzhang16/step
Google STEP