Getting started
Install the gateway and get a first completion through the router.
operator documentation
llm-router docs
Self-hosted AI gateway -- operator documentation
v1.1.6
llm-router-plugins v0.1.1 @ b7b68ce llm-router-services v0.0.4 @ d1b9641
llm-router
Configuration
Environment variables and the models.json routing configuration.
Routing & resilience
Load balancing strategies, failover and connection keep-alive.
Security & auditing
Authentication, authorization, rate limiting and the GPG audit trail.
Observability
Prometheus metrics exposed by the router and how to scrape them.
REST API
Endpoint catalogue and the guide for adding your own endpoints.
CLI
llm-router command line reference: server lifecycle, models, utilities.
Python library
Embed the router in your own code with llm_router_lib.
Deployment
Helm chart, container image and the runtime knobs that matter in production.
Examples
Runnable stacks: vLLM, Ollama, LangChain, LlamaIndex and client SDKs.
- Integration Examples with LLM Router
- Full Stack with Local Models
- Using LlamaIndex with Local Models via the LLM‑Router
- vLLM + google/gemma‑3‑12b‑it – Quick‑Start Guide (Ubuntu)
- vLLM + speakleash/Bielik-11B-v2.3-Instruct – Przewodnik Szybkiego Startu (Ubuntu)
Release notes
What changed in every released version.
llm-router-plugins
Plugins overview
The llm-router-plugins package: anonymizers, guardrails, semantic routing and RAG plugins.
Masker plugins
Built-in and custom PII anonymizer plugins.
Routing plugins
Semantic routing, bi-encoder routing and Codex CLI routing.
Utility plugins
RAG support and other pipeline utilities.
llm-router-services
Services overview
The llm-router-services HTTP API: guardrails, maskers and the unified Flask application.
Guardrail services
NASK-PIB, Sojka and dynamic guardrail routing endpoints.
Docker & deployment
Building, running and deploying the services container.