Hugging Face recommender is a utility package to estimate and get helpful information for deploying and training hugging face models. The package includes:
recommender
: python library to estimate the required resources for a given respecting mode memory, kv-cache and generation.api
: FastAPI app to expose the recommender as a REST API.gradio
: Gradio app to expose the recommender as a web app.notebooks
: Jupyter notebooks to test and experiment with the recommender, used for our partners, e.g. AWS, Cloudflare, etc.tests
: Unit tests for the recommender, not much here.
Summary: Returns Hugging Face Text Generation Configuration
Query Parameters:
model_id
(string, optional): Model IDgpu_memory
(integer, optional, default=0): GPU memorynum_gpus
(integer, optional, default=1): Number of GPUs
Summary: Generates an Instance Type recommendation for a given provider (gcp, huggingface) and returns the instance type and TGI configuration.
Path Parameters:
provider
(string, required): Provider name (one of:huggingface
,gcp
,aws
(sagemaker))
Query Parameters:
model_id
(string, optional): Model IDgpu_memory
(integer, optional, default=999): GPU memory
- Install the requirements:
pip install -e ".[api]"
- Run the app with in-memory cache:
cd api && uvicorn app.main:app --reload
- Build the app
docker build -t recommender-api -f api/Dockerfile .
- Run the app:
docker run -p 8000:8000 recommender-api
- curl the app:
To test the dummy route, you can use curl or any HTTP client:
curl -X GET 'localhost:8000/v1/provider/gcp/recommend?model_id=HuggingFaceH4%2Fzephyr-7b-beta'
Push to main and ArgoCD will deploy the app to the cluster.