Cog is an open-source tool that lets you package machine learning models in a standard, production-ready container. vLLM is a fast and easy-to-use library for LLM inference and serving.
You can deploy your packaged model to your own infrastructure, or to Replicate.
-
๐ Run vLLM in the cloud with an API. Deploy any vLLM-supported language model at scale on Replicate.
-
๐ญ Support multiple concurrent requests. Continuous batching works out of the box.
-
๐ข Open Source, all the way down. Look inside, take it apart, make it do exactly what you need.
Go to replicate.com/replicate/vllm and create a new vLLM model from a supported Hugging Face repo, such as google/gemma-2b
Important
Gated models require a Hugging Face API token,
which you can set in the hf_token
field of the model creation form.
![Create a new vLLM model on Replicate](https://private-user-images.githubusercontent.com/7659/336538103-a8f31837-0ed3-40f7-974c-d0a16ae48350.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3NzI2MzYsIm5iZiI6MTcyNDc3MjMzNiwicGF0aCI6Ii83NjU5LzMzNjUzODEwMy1hOGYzMTgzNy0wZWQzLTQwZjctOTc0Yy1kMGExNmFlNDgzNTAucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjdUMTUyNTM2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NDJiMGEyOTYwOTRjNGM4YzUwZDFjY2IzNGM4ZDg5NzM3MTM1YjIwMWU2ODcxY2E3ZjM3ZWJjY2UxODU5Mjg3YSZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.nf-MO33ZFuCppSOusra_CDhJmK1iE_Jqk7-l2ENfuSA)
Replicate downloads the model files, packages them into a .tar
archive,
and pushes a new version of your model that's ready to use.
![Trained vLLM model on Replicate](https://private-user-images.githubusercontent.com/7659/336537381-ebb84e12-9173-4fb0-8749-7293a105cf13.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3NzI2MzYsIm5iZiI6MTcyNDc3MjMzNiwicGF0aCI6Ii83NjU5LzMzNjUzNzM4MS1lYmI4NGUxMi05MTczLTRmYjAtODc0OS03MjkzYTEwNWNmMTMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjdUMTUyNTM2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YzI5YzY4MWY2ODc5ZmQ2NDJkY2NjZWVlNDdmZGNkY2E4Zjk3MWNmN2ZhYmZiNmViYzkwOTgzNDk0MWFmNGIwYiZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.xYU1BW-9rObqICzaxdzSfPNp0iPfyuX7Nf_qeBBhz_0)
From here, you can either use your model as-is, or customize it and push up your changes.
If you're on a machine or VM with a GPU, you can try out changes before pushing them to Replicate.
Start by installing or upgrading Cog. You'll need Cog v0.10.0-alpha11:
$ sudo curl -o /usr/local/bin/cog -L "https://github.com/replicate/cog/releases/download/v0.10.0-alpha11/cog_$(uname -s)_$(uname -m)"
$ sudo chmod +x /usr/local/bin/cog
Then clone this repository:
$ git clone https://github.com/replicate/cog-vllm
$ cd cog-vllm
Go to the Replicate dashboard and navigate to the training for your vLLM model. From that page, copy the weights URL from the Download weights button.
![Copy weights URL from Replicate training](https://private-user-images.githubusercontent.com/7659/336550042-97c403a9-ec49-418a-a7e2-b37cb0e0bb8c.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjQ3NzI2MzYsIm5iZiI6MTcyNDc3MjMzNiwicGF0aCI6Ii83NjU5LzMzNjU1MDA0Mi05N2M0MDNhOS1lYzQ5LTQxOGEtYTdlMi1iMzdjYjBlMGJiOGMucG5nP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDgyNyUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA4MjdUMTUyNTM2WiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9NGUwZjBhMjVlNGFmY2UxOWU0ODYzMWViODlhYmZhZjA4Y2ZmMjM2OTI0ZTBiZGIyYmU3Yzk0MTI0ZTdhMDc1MCZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.OMxrf6i5GUe0Po3O48eiiN2QeSjJbF2XNZfw3aIfOPI)
Set the COG_WEIGHTS
environment variable with that copied value:
$ export COG_WEIGHTS="..."
Now, make your first prediction against the model locally:
$ cog predict -e "COG_WEIGHTS=$COG_WEIGHTS" \
-i prompt="Hello!"
The first time you run this command,
Cog downloads the model weights and save them to the models
subdirectory.
To make multiple predictions,
start up the HTTP server and send it POST /predictions
requests.
# Start the HTTP server
$ cog run -p 5000 -e "COG_WEIGHTS=$COG_WEIGHTS" python -m cog.server.http
# In a different terminal session, send requests to the server
$ curl http://localhost:5000/predictions -X POST \
-H 'Content-Type: application/json' \
-d '{"input": {"prompt": "Hello!"}}'
When you're finished working, you can push your changes to Replicate.
Grab your token from replicate.com/account and set it as an environment variable:
export REPLICATE_API_TOKEN=<your token>
$ echo $REPLICATE_API_TOKEN | cog login --token-stdin
$ cog push r8.im/<your-username>/<your-model-name>
--> ...
--> Pushing image 'r8.im/...'
After you push your model, you can try running it on Replicate.
Install the Replicate Python SDK:
$ pip install replicate
Create a prediction and stream its output:
import replicate
model = replicate.models.get("<your-username>/<your-model-name>")
prediction = replicate.predictions.create(
version=model.latest_version,
input={ "prompt": "Hello" },
stream=True
)
for event in prediction.stream():
print(str(event), end="")