Topic: llm-serving Goto Github

👇 Here are 58 public repositories matching this topic...

alibaba / rtp-llm
llm-serving,RTP-LLM: Alibaba's high-performance LLM inference engine for diverse applications.
Organization: alibaba
gpt inference llama llm llm-serving llmops model-serving

antoniogr7 / pratical-llms
llm-serving,A collection of hand on notebook for LLMs practitioner
User: antoniogr7
genai llm llm-evaluation llm-inference llm-serving llm-training quantization

asprenger / ray_vllm_inference
llm-serving,A simple service that integrates vLLM with Ray Serve for fast and scalable LLM serving.
User: asprenger
inference llm llm-serving llmops mlops model-serving pytorch ray transformer vllm

azminewasi / awesome-llms-iclr-24
llm-serving,It is a comprehensive resource hub compiling all LLM papers accepted at the International Conference on Learning Representations (ICLR) in 2024.
User: azminewasi
large-language-model large-language-models large-language-models-and-translation-systems large-language-models-for-graph-learning llm llm-agent llm-evaluation llm-framework llm-inference llm-privacy

bentoml / bentoml
llm-serving,The easiest way to serve AI apps and models - Build reliable Inference APIs, LLM apps, Multi-model chains, RAG service, and much more!
Organization: bentoml
Home Page: https://bentoml.com
model-serving mlops llmops generative-ai llm-inference model-inference-service inference-platform deep-learning llm-serving machine-learning

bentoml / openllm
llm-serving,Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.
Organization: bentoml
Home Page: https://bentoml.com
llm llmops model-inference falcon fine-tuning stablelm llm-serving llama mpt vicuna

centml / llm-inference-bench
llm-serving,Lightweight and extensible LLM Inference serving benchmark tool written in Rust.
Organization: centml
benchmarking llm-inference llm-serving

chenhunghan / ialacol
llm-serving,🪶 Lightweight OpenAI drop-in replacement for Kubernetes
User: chenhunghan
ai cloudnative cuda ggml gptq gpu helm kubernetes langchain llamacpp llm llm-inference llm-serving openai python

diverged / tavily-go
llm-serving,An unofficial Go port of the official Tavily API Python Wrapper.
User: diverged
rag answer-engine chatgpt llm-serving retrieval-augmented-generation search-engine

ehsanghaffar / llm-practice
llm-serving,A self-hosted personal chatbot API with FastAPI. It allows you to interact with the Llama2 LLM (and other open-source LLMs) to have natural language conversations, generate text, and perform various language-related tasks.
User: ehsanghaffar
langchain llama2 llamacpp llm-serving

embeddedllm / embeddedllm
llm-serving,EmbeddedLLM: API server for Embedded Device Deployment. Currently support CUDA/OpenVINO/IpexLLM/DirectML/CPU
Organization: embeddedllm
llm llm-inference directml directx-12 gemma llama mistral phi-3 llm-serving model-inference

fork123aniket / llm-rag-powered-qa-app
llm-serving,A Production-Ready, Scalable RAG-powered LLM-based Context-Aware QA App
User: fork123aniket
context-aware-system eleutherai fine-tuning large-language-models llm-inference llm-serving llm-training llmops parameter-efficient-fine-tuning question-answering ray ray-serve retrieval-augmented-generation

france-travail / benchmark_llm_serving
llm-serving,A library to benchmark LLMs via their API exposure
Organization: france-travail
benchmark llm llm-serving vllm

france-travail / happy_vllm
llm-serving,A REST API for vLLM, production ready
Organization: france-travail
Home Page: https://france-travail.github.io/happy_vllm/
api-rest llm llm-serving production vllm

friendliai / friendli-client
llm-serving,Friendli: the fastest serving engine for generative AI
Organization: friendliai
Home Page: https://friendli.ai
generative-ai llm llm-inference llmops serving gpt gpt3 inference llama2 llm-serving

galeselee / awesome_llm_system-paperlist
llm-serving,Since the emergence of chatGPT in 2022, the acceleration of Large Language Model has become increasingly important. Here is a list of papers on accelerating LLMs, currently focusing mainly on inference acceleration, and related works will be gradually added in the future. Welcome contributions!
User: galeselee
Home Page: https://galeselee.gitbook.io/awesome-papers/
llm-inference llm-serving paperlist papers system

genia-dev / vibraniumdome-docs
llm-serving,LLM Security Platform Docs
Organization: genia-dev
Home Page: https://docs.vibraniumdome.com
adverarial-attacks chatgpt large-language-model llm llm-agent llm-eval llm-evaluation llm-firewall llm-framework llm-inference

helixml / helix
llm-serving,Multi-node production AI stack. Run the best of open source AI easily on your own servers. Create your own AI by fine-tuning open source models. Integrate LLMs with APIs. Run gptscript securely on the server
Organization: helixml
Home Page: https://tryhelix.ai
api finetuning function-calling golang gptscript helm k8s llama llm llm-agent llm-serving mistral mixtral openai openapi rag sdxl self-hosted stable-diffusion swagger

hpcaitech / swiftinfer
llm-serving,Efficient AI Inference & Serving
Organization: hpcaitech
Home Page: https://hpc-ai.com/
artificial-intelligence deep-learning gpt inference llama llama2 llm-inference llm-serving

hpmll / burstgpt
llm-serving,A ChatGPT(GPT-3.5) & GPT-4 Workload Trace to Optimize LLM Serving Systems
Organization: hpmll
dataset llm mlsys llm-serving

interestinglsy / swiftllm
llm-serving,A tiny yet powerful LLM inference system tailored for researching purpose. vLLM-equivalent performance with only 2k lines of code (2% of vLLM).
User: interestinglsy
cuda gpt inference inference-engine llama llm llm-inference llm-serving llmops mlops model-serving pytorch transformer transformers

ivanlulyf / bunny-llm
llm-serving,Deno LLM API Service
User: ivanlulyf
Home Page: https://bunny-llm.deno.dev
chatgpt chatgpt-api cloudflare-workers-ai llm llm-serving

ivynya / illm
llm-serving,internet llm - access your ollama (or any other local llm) instance from across the internet
User: ivynya
llm-serving ollama ollama-interface

jason-cs18 / hetserve-llms
llm-serving,A Overview of Efficiently Serving Large Language Models across Edge Devices
User: jason-cs18
llm-serving survey

liguodongiot / llm-action
llm-serving,本项目旨在分享大模型相关技术原理以及实战经验。
User: liguodongiot
Home Page: https://www.zhihu.com/column/c_1456193767213043713
llm llm-inference llm-serving llm-training llmops

loopglitch26 / hinglish-ai-mentor
llm-serving,Hinglish Chatbot powered by Azure Cognitive Services, Google Translate and Open AI
User: loopglitch26
Home Page: https://hinglish-chatbot-loopglitch26.streamlit.app/
azure google llm-serving nlp-machine-learning prompt-engineering

mani-kantap / llm-inference-solutions
llm-serving,A collection of all available inference solutions for the LLMs
User: mani-kantap
llm-inference llm-serving llmops

mddunlap924 / llm-inference-serving
llm-serving,This repository demonstrates LLM execution on CPUs using packages like llamafile, emphasizing low-latency, high-throughput, and cost-effective benefits for inference and serving.
User: mddunlap924
deepspeed large-language-models llamacpp llamafile llm-inference llm-serving llms vllm

microsoft / aici
llm-serving,AICI: Prompts as (Wasm) Programs
Organization: microsoft
ai rust wasm wasmtime inference language-model llm llm-framework llm-inference llm-serving

mosecorg / mosec
llm-serving,A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine
Organization: mosecorg
Home Page: https://mosec.readthedocs.io/
model-serving deep-learning machine-learning nerual-network mlops machine-learning-platform hacktoberfest gpu python pytorch

neural-dragon-ai / cynde
llm-serving,A Framework For Intelligence Farming
Organization: neural-dragon-ai
autoscaling llm-inference llm-serving modal-labs openai-api polars pydantic xgboost intelligence-farming pydantic-logfire

oscinis-com / awesome-llm-productization
llm-serving,Awesome-LLM-Productization: a curated list of tools/tricks/news/regulations about AI and Large Language Model (LLM) productization
Organization: oscinis-com
ai large-language-models llm llm-inference llm-serving llmops machine-learning mlops production transformer

predibase / lorax
llm-serving,Multi-LoRA inference server that scales to 1000s of fine-tuned LLMs
Organization: predibase
Home Page: https://loraexchange.ai
fine-tuning gpt llama llm llm-inference llm-serving llmops lora model-serving pytorch transformers

ray-project / anyscale-berkeley-ai-hackathon
llm-serving,Ray and Anyscale for UC Berkeley AI Hackathon!
Organization: ray-project
Home Page: https://ai.calhacks.io/
anyscale berkeley-ai hackathon llm llm-inference llm-serving ray-distributed

ray-project / llm-application
llm-serving,
Organization: ray-project
anyscale llm llm-serving ray-distributed nlp scalable-machine-learning

ray-project / llms-in-prod-workshop-2023
llm-serving,Deploy and Scale LLM-based applications
Organization: ray-project
Home Page: https://home.mlops.community/public/events/ray-workshop-2023-06-15
anyscale llm llm-inference llm-serving llms ray

ray-project / ray
llm-serving,Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
Organization: ray-project
Home Page: https://ray.io
ray distributed parallel machine-learning reinforcement-learning deep-learning python rllib hyperparameter-search optimization

ray-project / ray-educational-materials
llm-serving,This is suite of the hands-on training materials that shows how to scale CV, NLP, time-series forecasting workloads with Ray.
Organization: ray-project
deep-learning distributed-machine-learning ray-distributed ray-tune ray ray-train ray-data ray-serve generative-ai llm

ray-project / ray-llm
llm-serving,RayLLM - LLMs on Ray
Organization: ray-project
Home Page: https://aviary.anyscale.com
distributed-systems large-language-models ray serving transformers llm llm-inference llm-serving llmops

rohan-paul / llm-finetuning-large-language-models
llm-serving,LLM (Large Language Model) FineTuning
User: rohan-paul
gpt-3 gpt3-turbo large-language-models llama2 llm llm-finetuning llm-inference llm-serving llm-training mistral-7b

skypilot-org / skypilot
llm-serving,SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.
Organization: skypilot-org
Home Page: https://skypilot.readthedocs.io
cloud-computing data-science deep-learning gpu hyperparameter-tuning machine-learning tpu job-queue job-scheduler cloud-management

slai-labs / get-beam
llm-serving,Run GPU inference and training jobs on serverless infrastructure that scales with you.
Organization: slai-labs
Home Page: https://beam.cloud
artificial-intelligence machine-learning python serverless cloud-computing cost-optimization data-science deep-learning distributed-computing gpu-acceleration

stosan / commentator
llm-serving,
User: stosan
Home Page: https://code-commentator.onrender.com/
generative-ai llm-serving

substratusai / runbooks
llm-serving,Finetune LLMs on K8s by using Runbooks
Organization: substratusai
Home Page: https://www.substratus.ai
kubernetes kubernetes-operator llm-inference llm-serving llm-training llmops ml-platform mlops

sugarcane-ai / sugarcane-ai
llm-serving,npm like package ecosystem for Prompts 🤖
Organization: sugarcane-ai
Home Page: https://sugarcaneai.dev
ai llm prompt-engineering framework llm-chain llm-framework llm-serving llm-training llmops next

sugarcane-ai / sugarcane-ai.github.io
llm-serving,
Organization: sugarcane-ai
Home Page: https://sugarcaneai.dev
llm llmops generative-ai framework workflow reusable typescript api prompt prompt-engineering

suleymansevimli / run-llm-model-locally
llm-serving,You can run any large language model on your local machine with this repository.
User: suleymansevimli
git-lfs huggingface llm llm-serving python

superduper-io / superduper
llm-serving,Superduper: Bring AI to your database! Integrate AI models and workflows with your database to implement custom AI applications, without moving your data. Including streaming inference, scalable model hosting, training and vector search.
Organization: superduper-io
Home Page: https://superduper.io
ai chatbot data database distributed-ml inference llm-inference llm-serving llmops ml mlops mongodb pretrained-models python pytorch rag semantic-search torch transformers vector-search

torchpipe / torchpipe
llm-serving,Serving Inside Pytorch
Organization: torchpipe
Home Page: https://torchpipe.github.io/
deployment inference pipeline-parallelism serving tensorrt triton-inference-server ray pytorch torch2trt serve

vllm-project / vllm
llm-serving,A high-throughput and memory-efficient inference and serving engine for LLMs
Organization: vllm-project
Home Page: https://docs.vllm.ai
gpt llm pytorch llmops mlops model-serving transformer llm-serving inference llama