A lightweight, OpenAI-compatible embedding & rerank service. Embedding aligns with OpenAI /v1/embeddings, rerank aligns with Cohere /v1/rerank.

📄 中文

On Apple Silicon (MPS), lightweight models deliver dramatically better throughput than heavy ones:

💡 Recommendation: Start with e5-small (default) for embeddings, bge-reranker-base for reranking. Both pre-loaded by python preload.py.

The service supports two rerank modes — cosine (embedding-based, instant) and cross-encoder (dedicated reranker, high accuracy). The examples below demonstrate the critical difference on the same query.

💡 Run ./compare_rerank.sh to reproduce these results on your own machine.

Query: "How to lose weight effectively?"

🔍 Cosine: all three scores cluster in a narrow range (0.78–0.88) — the model struggles to penalize the irrelevant Bitcoin doc because it shares surface-level features with the other texts. 🎯 Cross-encoder: score spread is 15.69 points — clearly separates relevant (+5.50) from irrelevant (-10.19), making top-1 selection unambiguous.

Query: "苹果公司最新产品是什么？" (What is Apple's latest product?)

🔍 Cosine ranks fruit above iPhone! The fruit doc (0.8720) outscores the correct answer (0.8455) — because cosine similarity treats all documents as bags of words and "苹果" (apple) appears heavily in both the query and the fruit doc. 🎯 Cross-encoder correctly places the iPhone announcement first, understanding that "苹果公司" (Apple Inc.) refers to the company, not the fruit.

💡 Recommendation: Use cosine for fast pre-filtering of large candidate sets, then cross-encoder for precise top-N reranking. The compare_rerank.sh script helps you evaluate which mode fits your use case.

{
  "model": "bge-m3",
  "input": "The quick brown fox jumps over the lazy dog",
  "encoding_format": "float",
  "dimensions": null
}

{
  "object": "list",
  "model": "bge-m3",
  "data": [
    {
      "object": "embedding",
      "index": 0,
      "embedding": [0.0123, -0.0456, 0.0789, "..."]
    }
  ],
  "usage": {
    "prompt_tokens": 10,
    "total_tokens": 10
  }
}

Dual-mode: cosine (lightweight, uses embedding model) or cross-encoder (professional reranker).

// Request: POST /v1/rerank
{
  "model": "bge-reranker-base",
  "query": "What is machine learning?",
  "documents": [
    "Deep learning is a subset of ML",
    "The weather is nice today",
    "Machine learning is a branch of AI"
  ]
}

// Response
{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "model": "bge-reranker-base",
  "results": [
    {"index": 2, "relevance_score": 0.98, "document": "Machine learning is a branch of AI"},
    {"index": 0, "relevance_score": 0.82, "document": "Deep learning is a subset of ML"},
    {"index": 1, "relevance_score": 0.12, "document": "The weather is nice today"}
  ],
  "meta": {"api_version": "1.0"}
}

⚠️ Run preload first! Downloads both embedding and reranker models (~1.2GB total). Skip this and the first API call will block until the download completes.

# 1. Download models (required — do this before starting!)
cd lite-emb
uv sync
python preload.py

# 2. Start the server
uv run python main.py

Optional: customize models or port in .env:

cp .env.example .env
# Edit .env to change MODEL_NAME, DEV_PORT, etc.

The service runs at http://localhost:8000. Visit /docs for interactive API docs.

# Health check
curl http://localhost:8000/health

# Service info (current model, dimension, etc.)
curl http://localhost:8000/

# Model list
curl http://localhost:8000/v1/models

# Single text embedding
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "bge-m3", "input": "Hello, world!"}'

# Batch embedding
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "bge-m3", "input": ["text one", "text two", "text three"]}'

# Rerank (cosine mode — instant)
curl -X POST http://localhost:8000/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"e5-small","query":"What is ML","documents":["Deep learning","Nice weather","AI branch"]}'

# Rerank (cross-encoder mode — higher accuracy)
curl -X POST http://localhost:8000/v1/rerank \
  -H "Content-Type: application/json" \
  -d '{"model":"bge-reranker-base","query":"What is ML","documents":["Deep learning","Nice weather","AI branch"]}'

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8000/v1",
    api_key="not-needed"  # lite-emb requires no API key
)

response = client.embeddings.create(
    model="bge-m3",
    input="Your text here"
)

print(response.data[0].embedding)

Any HuggingFace sentence-transformers compatible model works out of the box — no pre-registration needed. The system will:

1. Check if it is a known model → use the registered backend
2. Model name contains bge-m3 → use BGEM3FlagModel backend
3. Everything else → use SentenceTransformer backend

# Use any HuggingFace model directly
curl -X POST http://localhost:8000/v1/embeddings \
  -H "Content-Type: application/json" \
  -d '{"model": "e5-base", "input": "hello"}'

# Download the default model (BAAI/bge-m3)
uv run python preload.py

# Download a specific model
uv run python preload.py --model sentence-transformers/all-MiniLM-L6-v2

# Start in offline mode after pre-downloading
HF_HUB_OFFLINE=1 uv run python main.py

Request arrives at POST /v1/embeddings
  ├─ Model already loaded? → Return cached instance
  ├─ Different model requested?
  │   ├─ ALLOW_MODEL_SWITCH=true → Unload old model → Load new model
  │   └─ ALLOW_MODEL_SWITCH=false → Return 400 error
  └─ First load?
      ├─ Check local cache
      ├─ Not cached → Download from HuggingFace (via hf-mirror)
      └─ Load via BGEM3FlagModel / SentenceTransformer into memory

# Build and start
docker compose build --build-arg USE_APT_MIRROR=true --build-arg USE_PYPI_MIRROR=true
docker compose up -d

# View logs
docker compose logs -f

# Stop
docker compose down

🔑 China users: Configure Docker Desktop → Settings → Resources → Proxies first, otherwise pulling python:3.11-slim-bookworm base image will timeout. The proxy only affects the Docker daemon pulling base images — build-stage apt/pip/model downloads still go direct to domestic mirrors.

All settings are managed via .env file or environment variables:

MIT