secure-inference is a gateway level access control system for llm-d. It provides JWT-based authentication and attribute-based access control (ABAC) for LLM inference requests, operating independently of llm-d internals. Learn More

This provides an Envoy [ext-auth] compatible gRPC server that sits in front of LLM-D inference pools. It validates JWT tokens, looks up users and models from Kubernetes CRDs, and evaluates access policies using Open Policy Agent (OPA) — all in a single binary.

For details on the internal structure, component patterns, and dependency layers, see the [Architecture Documentation].

Admins define User and Model CRDs. Kubernetes controllers sync these into an in-memory store. When a request arrives, the ext-auth server validates the JWT, looks up the user and model, and asks the OPA policy engine whether access is allowed. Optionally, for base model requests, a Python sidecar selects the best LoRA adapter via semantic similarity.

See User and Model CRDs for the full CRD reference and access policy details.

• Go 1.24+
• Docker (for container builds)
• pre-commit (for local development)

# Clone the repo
git clone https://github.com/llm-d-incubation/secure-inference.git
cd secure-inference

# Install pre-commit hooks
pre-commit install

# Build
make build

# Run tests
make test

# Run linters
make lint

make help           # Show all available targets
make build          # Build secure-inference binary
make build-all      # Build all binaries (main + CLI + deployment-customizer)
make test           # Run unit tests
make test-e2e       # Run e2e tests
make lint           # Run Go and Python linters
make fmt            # Format Go and Python code
make image-build    # Build Docker images
make pre-commit     # Run pre-commit hooks
make deploy         # Deploy all components to cluster

For local development and deployment, see the tutorial on deploying a multi-tenant AI inference stack with LoRA adapters and attribute-based access control - no GPU required

• Architecture Documentation
• ext-auth
• Create an issue

See CONTRIBUTING.md for development guidelines, coding standards, and how to submit changes.

We welcome contributions! Please see CONTRIBUTING.md for guidelines.

All commits must be signed off (DCO). See PR_SIGNOFF.md for instructions.

For large changes please [create an issue] first describing the change so the maintainers can do an assessment.

To report a security vulnerability, please see SECURITY.md.

This project is licensed under the Apache License 2.0 - see LICENSE for details.