LLM Router - Open-Source AI Gateway for Local and Cloud LLM Infrastructure#

LLM Router is a service that can be deployed on‑premises or in the cloud. It adds a layer between any application and the LLM provider. In real time it controls traffic, distributes load among providers of a specific LLM, and enables analysis of outgoing requests from a security perspective (masking, anonymization, prohibited content). It is an open‑source solution (Apache 2.0) that can be launched instantly by running a ready‑made image in your own infrastructure.

🌐 Ecosystem Overview#

The LLM‑Router project is split across five dedicated repositories:

✨ Key Features#

🧩 Plugin System Architecture#

LLM Router uses a registry-based pipeline pattern. Each plugin implements a tiny, well‑defined apply method and can be composed in an ordered list to form a pipeline. Pipelines are instantiated by the MaskerPipeline, GuardrailPipeline and UtilsPipeline classes and are driven automatically by the endpoint logic in endpoint_i.py.

Data flow#

Request → MaskerPipeline → GuardrailPipeline → UtilsPipeline → Model Provider

Masker Plugins#

Guardrail Plugins#

Utility Plugins#

Pipelines are configured via environment variables:

# Comma-separated list of masker plugins to apply
export LLM_ROUTER_MASKING_STRATEGY_PIPELINE="fast_masker,pii_masker"

# Enable guardrails
export LLM_ROUTER_FORCE_GUARDRAIL_REQUEST=1

# Enable masking entirely
export LLM_ROUTER_FORCE_MASKING=1

# Record masking operations in audit log
export LLM_ROUTER_MASKING_WITH_AUDIT=1

Semantic BiEncoder Routing#

The semantic_biencoder_routing plugin uses a neural embedding model (google/embeddinggemma-300m) to compute semantic embeddings for a set of pre‑configured routing targets. Each target has a name, a model_name (the model to route to), a description, and a list of examples. At query time the user message is embedded and matched against all stored target embeddings using FAISS (IndexFlatIP on L2‑normalised vectors = cosine similarity). The best‑matching target determines the selected model.

How it works#

1. Index building (on first load or when the persist directory is missing):

• For each routing target, its description and examples are combined into text.
• The text is split into overlapping token chunks using a sliding window (chunk_size tokens, chunk_overlap tokens overlap).
• Each chunk is embedded via the BiEncoder model (e.g. google/embeddinggemma-300m).
• All embedding vectors are L2‑normalised to unit length.
• Vectors are inserted into a faiss.IndexFlatIP index (inner product).
• A docstore maps each FAISS document ID to its target name (for reverse lookup).

2. Routing (query):

• The user message is embedded and L2‑normalised.
• FAISS performs a nearest‑neighbor search returning the top_k closest chunks.
• Scores are aggregated per target: the mean cosine similarity of all chunks belonging to the same target is computed.
• The target with the highest mean similarity wins and its model_name is returned.

3. Persistence:

The FAISS index and docstore are saved to disk (files index.faiss and docstore.pkl) under the configured persist directory. On subsequent starts the index is loaded from disk — embeddings are not recomputed. If the embedding model changes (different output dimension) the index is automatically rebuilt.

Full example JSON config with routing_targets and their examples is available in the plugins repo: llm_router_plugins/resources/routing/semantic_biencoder.json. Detailed variable descriptions and usage examples: Plugin Routing README.

Example routing targets:

📦 Quick Start#

1️⃣ Create & activate a virtual environment#

python3 -m venv .venv
source .venv/bin/activate

# Only the core library (llm-router-lib).
pip install .

# Core library + API wrapper (llm-router-api).
pip install .[api]

# Core library + API wrapper + Prometheus metrics.
pip install .[api,metrics]

Note: When Prometheus metrics are enabled, LLM_ROUTER_USE_PROMETHEUS=1 must be set and Redis is required (used for provider availability state).
The multiproc directory defaults to $HOME/.llm-router/metrics/prometheus/multiproc — override via the PROMETHEUS_MULTIPROC_DIR environment variable if needed.

Then start the application with the environment variable set:

export LLM_ROUTER_USE_PROMETHEUS=1

When LLM_ROUTER_USE_PROMETHEUS is enabled, the router automatically registers a /metrics endpoint (under the API prefix, e.g. /api/metrics). This endpoint exposes Prometheus‑compatible metrics such as request counts, latencies, and any custom counters defined by the application.

2️⃣ Run the REST API#

./run-rest-api.sh
# or
LLM_ROUTER_MINIMUM=1 python3 -m llm_router_api.rest_api

3️⃣ Quick‑start guides for local models#

• Gemma 3 12B‑IT – README
• Bielik 11B‑v2.3‑Instruct – README

4️⃣ Integration boilerplates#

Integration examples for popular LLM libraries (LlamaIndex, LangChain, OpenAI, LiteLLM, Haystack) are in the examples/ directory. See examples README for details.

🔐 Auditing#

The router can record request‑level events (guard‑rail checks, payload masking, custom logs) in a tamper‑evident, encrypted form. All audit entries are written by the auditor module and stored under logs/auditor/ as GPG‑encrypted files.

For a complete guide — including key generation, encryption workflow, and decryption utilities — see:

➡️ Auditing subsystem documentation

Utility scripts:

• scripts/gen_and_export_gpg.sh — generate and export GPG keys
• scripts/decrypt_auditor_logs.sh — decrypt encrypted audit logs

🔐 Authentication#

The router supports API-key-based authentication with per-endpoint policies, rate limiting, and audit trail.

➡️ Authentication documentation

⏱️ Rate Limiting#

Sliding-window rate limiting backed by Redis sorted sets. Each API key + IP gets a configurable number of requests per minute. Returns Retry-After on 429 responses and exposes Prometheus metrics.

➡️ Rate Limiting documentation

🖥️ CLI Reference#

The llm-router package provides a command-line tool for managing API keys, policies, rate-limit presets, and anonymizing text. A full command reference is available here:

➡️ CLI Command Reference

🔒 Security#

🔍 Error message sanitization#

All error messages returned to API callers are sanitized to prevent leakage of internal infrastructure details (IP addresses, hostnames, URLs, ports, connection strings).

How it works:

• sanitize_error_message() in llm_router_api/core/errors.py strips URLs, IP addresses, ports, hostnames, and urllib3/requests exception internals from error strings.
• Applied at every output choke point:
  • HTTP provider errors (httprequest.py)
  • Streaming error chunks (stream_handler.py)
  • return_response_not_ok() — the central error builder for all non-streaming errors
  • Parameter validation errors in register.py
• Server-side logs still receive the full, unsanitized exception — debugging remains fully possible.

What you will see as a caller:

• ✅ "ConnectTimeout: The read operation timed out"
• ✅ "A connection error occurred"

What you will NOT see:

• ❌ 192.168.x.x, 10.0.x.x — internal IPs
• ❌ http://..., https://... — internal URLs
• ❌ port=8080, host='...' — connection details
• ❌ Stack traces or internal provider addresses

This protection applies to all error responses regardless of whether they originate from HTTP provider calls, streaming endpoints, or request validation.

📦 Docker#

Run the container with the default configuration:

docker run -p 5555:8080 quay.io/radlab/llm-router:rc1

For more advanced usage you can use a custom launch script:

#!/bin/bash

PWD=$(pwd)

docker run \
  -p 5555:8080 \
  -e LLM_ROUTER_TIMEOUT=500 \
  -e LLM_ROUTER_IN_DEBUG=1 \
  -e LLM_ROUTER_MINIMUM=1 \
  -e LLM_ROUTER_EP_PREFIX="/api" \
  -e LLM_ROUTER_SERVER_TYPE=gunicorn \
  -e LLM_ROUTER_SERVER_PORT=8080 \
  -e LLM_ROUTER_SERVER_WORKERS_COUNT=4 \
  -e LLM_ROUTER_DEFAULT_EP_LANGUAGE="pl" \
  -e LLM_ROUTER_LOG_FILENAME="llm-proxy-rest.log" \
  -e LLM_ROUTER_EXTERNAL_TIMEOUT=300 \
  -e LLM_ROUTER_BALANCE_STRATEGY=balanced \
  -e LLM_ROUTER_REDIS_HOST="192.168.100.67" \
  -e LLM_ROUTER_REDIS_PORT=6379 \
  -e LLM_ROUTER_MODELS_CONFIG=/srv/cfg.json \
  -e LLM_ROUTER_PROMPTS_DIR="/srv/prompts" \
  -v "${PWD}/resources/configs/models-config.json":/srv/cfg.json \
  -v "${PWD}/resources/prompts":/srv/prompts \
  quay.io/radlab/llm-router:rc1

Kubernetes (Helm)#

Helm charts for Kubernetes deployment are available in the helm_charts/ directory.

🛠️ Configuration (via environment)#

A full list of environment variables is available at: API README

Core variables#

Masking & guardrail variables#

Redis variables#

Note: When LLM_ROUTER_REDIS_HOST is set, the router uses Redis for load‑balancing state and provider availability tracking.

Semantic BiEncoder Routing variables#

LangChainRAG variables#

Note: LangChainRAG requires the llm-router-plugins package. When LLM_ROUTER_UTILS_PLUGINS_PIPELINE includes langchain_rag, these variables configure the RAG plugin behavior.

Plugin Pipeline variables#

Note: Utility plugins run per‑request in the pipeline between endpoint processing and model provider dispatch. Available plugins: simple_semantic_routing, semantic_biencoder_routing, langchain_rag. Each plugin has additional configuration documented above.

Authentication variables#

Note: Rate limiting is always applied when authentication is enabled — there is no separate toggle for it. Auth Redis (LLM_ROUTER_AUTH_REDIS_*) is independent from general Redis (LLM_ROUTER_REDIS_*).

See full authentication docs: llm_router_api/AUTHENTICATION.md

⚖️ Load Balancing Strategies#

The current list of available strategies, the interface description, and an example extension can be found at: Load‑Balancing Strategies

Strategies: balanced, weighted, dynamic_weighted, first_available, first_available_optim.

🛣️ Endpoints Overview#

The list of endpoints — categorized into built‑in, provider‑dependent, and utility endpoints — and a description of the streaming mechanisms can be found at: Endpoints Overview

Highlights#

Note: By default LLM_ROUTER_AUTH_ENABLED=false, so all endpoints are accessible without authentication. Set it to "true" to enforce auth. The _public list (default /ping,/version,/models,/) can be customized via LLM_ROUTER_AUTH_PUBLIC_ENDPOINTS.

🌐 Web Applications#

Config Manager (port 8081)#

Full web UI for managing LLM Router model configurations:

• Multi‑user with authentication and role‑based access (admin/user)
• Projects — group configurations by project
• Model configuration — create, edit, import/export JSON configs; manage providers across families (Google, OpenAI, Qwen)
• Version control — snapshot history with restore capability
• Active model selection — choose which models to activate per config
• Drag‑and‑drop provider reordering (HTMX)
• Light/dark themes (Alpine.js)
• 26+ API endpoints under /configs

Run: ./run-configs-manager.sh

Anonymizer (port 8082)#

Web UI for text anonymization and interactive chat:

• 3 anonymization algorithms: fast (regex), pii_masking (ML model), fast+pii (hybrid)
• Interactive chat with streaming SSE responses and session persistence
• Dynamic model selection from the router
• i18n — Polish and English translations (122 keys)
• Privacy warnings when anonymization is disabled
• Privacy policy & terms pages

Run: ./run-anonymizer.sh

🧰 llm-router-utils#

The llm-router-utils repository provides CLI tools and ready‑made deployment configs:

CLI Tools#

Speakleash Deployment Configs#

The resources/llm-router-speakleash/ directory contains ready‑made configs for deploying Speakleash models:

• speakleash-models.json — configures Bielik-11B-v2.3-Instruct across 8 vLLM providers on 3 hosts
• run-bielik-*.sh — vLLM launch scripts for each GPU (cuda:0, cuda:1, cuda:2)
• run-rest-api-gunicorn.sh — full LLM Router server with masking, guardrails, Redis balancing, and Prometheus metrics
• run-sojka-guardrail.sh — guardrail service with Bielik‑Guard model

⚙️ Configuration Details#

🔧 Development#

• Python 3.10+ (project is tested on 3.10.6)
• All dependencies are listed in requirements.txt. Install them inside the virtualenv.
• To add a new provider, create a class in llm_router_api/core/api_types that implements the BaseProvider interface and register it in llm_router_api/register/__init__.py.

📚 Changelog#

See the CHANGELOG for a complete history of changes.

📜 License#

See the LICENSE file.