This repository simply downloads the Mistral 7B model and runs it as a local server.
- Docker
Make sure you have docker
and docker-compose
on your machine. The model is wrapped in a llamafile distributable.
-
Clone the repository
git clone [email protected]:mroyal/llm-server.git llm-server cd llm-server
-
Download the model
Optionally, download the Mistral 7B model and put it in the
models
directory inside the repository. If you download it, it will be saved to your Docker image. Otherwise, each container will attempt the download as your service boots.wget https://huggingface.co/jartine/mistral-7b.llamafile/resolve/main/mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile -O models/mistral-7b-instruct-v0.1-Q4_K_M-server.llamafile
-
Start the service
Docker:
docker-compose up --build
Kubernetes:
kubectl apply -f k8s
-
Profit
Navigate to http://127.0.0.1:8080/ (Docker) or http://127.0.0.1:PORT (retrieve PORT from
kubectl get services
) and have fun!