All environment variables share the LLM_ROUTER_ prefix. They are loaded from os.environ in llm_router_api/base/constants.py at import time and validated by
_StartAppVerificator.
Lifetime of one held worker slot in first_available_optim_nworkers. Renewed while the router process lives, so it bounds how long a crashed process keeps occupying a provider. Must exceed the longest expected request only if the KeepAlive monitor cannot keep up.
LLM_ROUTER_LB_SLOT_MAX_AGE_SECONDS
1800
How long an unreleased worker slot keeps being renewed, counted from the acquisition. Covers a request that never released its slot (an abandoned stream): it is left to expire instead of occupying the provider until the router restarts. 0 or negative disables the cap.
LLM_ROUTER_LB_HOST_PIN_TTL_SECONDS
3600
Lifetime of the “this host serves this model” pin used by first_available_optim / first_available_optim_nworkers (Redis key host:<host>). Refreshed on every selection, so only a host nothing has selected for this long becomes available to another model.
LLM_ROUTER_SERVER_TYPE
flask
Server implementation: flask, gunicorn, waitress.
LLM_ROUTER_SERVER_PORT
8080
Port on which the server listens.
LLM_ROUTER_SERVER_HOST
localhost
Host address for the server.
LLM_ROUTER_SERVER_WORKERS_COUNT
2
Number of workers (for servers that support them).
LLM_ROUTER_SERVER_THREADS_COUNT
8
Number of worker threads (for servers that support them).
LLM_ROUTER_SERVER_WORKER_CLASS
None
Worker class for servers that support it (e.g. gevent).
See also PROMETHEUS_MULTIPROC_DIR for the directory where Prometheus multiprocess worker data files are stored.
LLM_ROUTER_VERBOSE is a debugging aid, not a logging level. With this variable on, SecureEndpointI stores it
as self._verbose_mode and run_ep writes the raw, unmasked request payload (JSON) to the log before masking,
so the log contains exactly the PII that masking would otherwise strip. On top of that, rest_api.main() emits a
WARNING and sleeps 3 s (VERBOSE_STARTUP_DELAY_SECONDS) before the server binds its port, so an accidental
production start can still be aborted. Enable it only for local troubleshooting (LLM_ROUTER_VERBOSE=1,
llm-router server start --verbose) and never in production or on shared log storage.
Relevant only for providers with api_type: "vertex_ai" that do not carry an explicit
api_token; the router then resolves a Google access token through Application Default
Credentials (service-account JSON via provider_options.credentials_file or
GOOGLE_APPLICATION_CREDENTIALS, GCE/GKE metadata server, Workload Identity). This needs
the optional google dependency (pip install "radlab-llm-router[google]", see
requirements-google.txt).
Variable
Default
Description
LLM_ROUTER_GOOGLE_SCOPES
https://www.googleapis.com/auth/cloud-platform
Comma‑separated OAuth2 scopes requested when resolving an access token.
LLM_ROUTER_GOOGLE_TOKEN_REFRESH_SKEW
120
Seconds before expiry at which a cached access token is refreshed. A malformed value falls back to 120 instead of breaking start‑up.
Token lifetime handling: google-auth reports credentials.expiry as a naive
UTC datetime, and the router reads it as UTC — so the cached lifetime no longer
shifts with the machine's timezone. Refreshes are serialised per credential
(a burst of concurrent requests mints one token, not one per request), and a
401/403 from Vertex evicts the cached token so the next request mints a
fresh one rather than replaying a rejected credential until the process restarts.
Requires the llm-router-plugins package. When LLM_ROUTER_UTILS_PLUGINS_PIPELINE includes
semantic_biencoder_routing, these variables configure the plugin:
Variable
Default
Description
LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_CONFIG
(empty)
Config source of truth — raw JSON string or file path. Falls back to bundled semantic_biencoder.json. Individual env vars below override loaded config values only when explicitly set.
LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_MODEL
(empty)
Override the embedding model name or local path.
LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_TARGETS
(empty)
Pipe-separated list of target names (overrides all targets).
Rate limiting is always applied when authentication is enabled:
Variable
Default
Description
LLM_ROUTER_AUTH_DEFAULT_RATE_LIMIT
60
Default rate limit (requests per minute). Rate limiting is always active when authentication is enabled — there is no separate toggle.
LLM_ROUTER_RATE_LIMITING_CONFIG
(empty)
Path to a rate-limit presets file or directory (rate-limiting.json) used by llm-router auth; empty falls back to the user config file, the packaged resource and the builtin presets.
Comma-separated IPs/CIDRs of trusted proxies. Only a direct peer listed here may supply X-Forwarded-For; otherwise the header is ignored (anti-spoofing for rate-limit/IP checks).
LLM_ROUTER_AUTH_FAILURE_LIMIT
20
Max failed-authentication attempts (missing/invalid key) per client IP per window before a 429 lockout. 0 disables the lockout.
Per-provider health-check ping timeout (seconds). Busy LLM hosts (e.g. Ollama serving a large model) may need more than 2 s to accept a new connection.
Number of consecutive failed pings required before a provider is marked unavailable (hysteresis, prevents a single slow ping from removing a live provider).
The definitive list of all runtime environment variables is in the source code: llm_router_api/base/constants.py. Each constant is loaded from os.environ with a
documented default and validated at import time.