{"version":"0.1.1","pages":[{"k":"index-2.html","t":"Plugins overview","s":"Plugins overview","x":"## Overview The **LLM‑Router** project ships with a modular plugin system that lets you plug‑in **anonymizers** (also called *maskers*) and **guardrails** into request‑processing pipelines. Each plugin implements a tiny, well‑defined inter…","h":["Overview","1. Anonymizers (Maskers)","1.1 What they do","1.2 Built‑in anonymizer plugins","1.3 How a masker is used","2. Guardrails","2.1 What they do","2.2 Built‑in guardrail plugins","2.3 How a guardrail is used","2.5 ML-Based PII Classification","How it complements regex maskers","Quick start","2.6 Polish Identification Regex Patterns","Available Polish rules","How they work","Adding rules to the pipeline","2.7 Semantic Routing (Model Selection)","2.7.1 Simple Semantic Routing (Heuristic)","2.7.2 Bi-Encoder Semantic Routing (Model Selection)","2.7.3 Codex CLI routing (auto_codex)","2.7.4 Claude Code Model Swap (claude-* → configured models)","3. Pipelines","3.1 Registration","3.2 Configuration","4. Adding a New Plugin","5. Retrieval‑Augmented Generation (RAG) Support","5.1 What the plugin does","5.2 Environment variables"],"b":"Overview # The LLM‑Router project ships with a modular plugin system that lets you plug‑in anonymizers (also called maskers ) and guardrails into request‑processing pipelines. Each plugin implements a tiny, well‑defined interface ( apply ) and can be composed in an ordered list to form a * pipeline *. Pipelines are instantiated by the MaskerPipeline and GuardrailPipeline classes and are driven automatically by the endpoint logic in endpoint_i.py . 1. Anonymizers (Maskers) # 1.1 What they do # Goal – Remove or replace personally‑identifiable information (PII) from a payload before it reaches the LLM or an external service. Typical strategy – Run a pipeline of maskers that locate spans corresponding to IDs, emails, IPs, etc., and replace each span with a placeholder such as {{MASKED_ITEM}} . 1.2 Built‑in anonymizer plugins # Plugin Description Technical notes FastMaskerPlugin ( fast_masker_plugin.py ) Thin wrapper around the FastMasker utility class. Receives a JSON‑compatible payload and returns the same payload with all detected PII masked. Implements PluginInterface . The heavy lifting is delegated to FastMasker.mask_payload(payload) . No extra I/O; the FastMasker instance is created once in __init__ . 1.3 How a masker is used # The endpoint (e.g. EndpointI._do_masking_if_needed ) checks the global flag FORCE_MASKING . If enabled, it creates a MaskerPipeline with the list of masker plugin identifiers (e.g. [\"fast_masker\"] ). The pipeline calls each plugin’s apply method sequentially, feeding the output of one as the input of the next. The final payload – now stripped of PII – proceeds to the rest of the request flow (guardrails, model dispatch, etc.). 2. Guardrails # 2.1 What they do # Goal – Verify that a request (or its response) complies with policy rules (e.g. no hateful, illegal, or unsafe content). Typical strategy – Split the payload into manageable text chunks, run a pipeline of guardrails, aggregate per‑chunk scores, and decide whether the overall request is safe. 2.2 Built‑in guardrail plugins # Plugin Description Technical notes NASKGuardPlugin ( nask_guard_plugin.py ) HTTP‑based guardrail that forwards the payload to the external NASK guardrail service ( /nask_guard endpoint) and returns a boolean safe flag together with the raw response. Inherits from HttpPluginInterface . The apply method calls _request(payload) (provided by the base class) and extracts results[\"safe\"] . Errors are caught and logged; on failure the plugin returns (False, {}) . SojkaGuardPlugin ( sojka_guard_plugin.py ) HTTP‑based guardrail that forwards the payload to the Sójka guardrail service ( /sojka_guard endpoint) and returns a safety flag. Mirrors the design of NASKGuardPlugin . The endpoint_url is built from the LLM_ROUTER_GUARDRAIL_SOJKA_GUARD_HOST environment variable. On success it returns (True, response) , otherwise (False, {}) . (Implicit) GuardrailProcessor ( processor.py ) Core logic used by the internal NASK guardrail Flask route ( nask_guardrail ). Tokenises the payload, creates overlapping chunks, runs a Hugging‑Face text‑classification pipeline, and produces a detailed safety report. Handles model loading ( AutoTokenizer , pipeline(\"text‑classification\") ), chunking ( _chunk_text ), and scoring thresholds ( MIN_SCORE_FOR_SAFE , MIN_SCORE_FOR_NOT_SAFE ). Returns a dict: {\"safe\": <bool>, \"detailed\": [...]} . 2.3 How a guardrail is used # The endpoint calls _is_request_guardrail_safe(payload) (or the analogous response guardrail). If FORCE_GUARDRAIL_REQUEST is true, a GuardrailPipeline is built from the configured plugin IDs (e.g. [\"nask_guard\", \"sojka_guard\"] ). The pipeline iterates over each guardrail plugin; each apply returns (is_safe, message) . The first plugin that reports is_safe=False short‑circuits the pipeline and the request is rejected with a 400/500 error payload. 2.5 ML-Based PII Classification # For cases where regex patterns alone are insufficient (e.g. context-dependent PII detection), the project integ…","r":"llm-router-plugins"},{"k":"llm-router-plugins/maskers/fast-masker/index.html","t":"Overview","s":"Masker plugins","x":"## Overview The **fast_masker** plugin provides a simple, rule‑based engine that scans a piece of text and replaces sensitive data ( e‑mail addresses, IPs, URLs, phone numbers, Polish PESEL identifiers, etc.) with clearly marked placeholde…","h":["Overview","Masking Rules","1. Highest Certainty — Checksum Validated Identifiers","2. High Certainty — Strict Format Validation","3. Medium-High Certainty — International Phone Numbers","4. Medium Certainty — Well-Structured Formats","5. Medium-Low Certainty — Business Identifiers","6. Lower Certainty — Pattern-Based with Context","7. Format-Based — Specific Patterns","8. Lowest Certainty — Generic Patterns","9. Beta Features","Disabled Rules (Too Noisy)","Utility Validators","Polish Identification Numbers","Financial Numbers","Vehicle and Transport","International Identification","Network and Security","Business Identifiers"],"b":"Overview # The fast_masker plugin provides a simple, rule‑based engine that scans a piece of text and replaces sensitive data ( e‑mail addresses, IPs, URLs, phone numbers, Polish PESEL identifiers, etc.) with clearly marked placeholders. The core component is the :class: ~llm_router_plugins.plugins.fast_masker.core.masker.FastMasker , which receives an ordered list of rule objects and applies each rule sequentially to the input text. Because the rules are applied in the order they are supplied, you can control precedence (e.g., replace URLs before e‑mails if needed). Masking Rules # Rules are applied in order from highest certainty (checksum-validated) to lowest certainty (pattern-based). This ordering minimizes false positives and ensures the most reliable identifiers are masked first. 1. Highest Certainty — Checksum Validated Identifiers # Rule Placeholder What it Detects Notes CreditCardRule {{CREDIT_CARD}} Credit card numbers (13-19 digits with optional spaces/dashes, e.g., 4532 1234 5678 9010 ). Validates using Luhn algorithm checksum. VinRule {{VIN}} Vehicle Identification Numbers (17 characters, e.g., 1HGBH41JXMN109186 ). Validates using ISO 3779 checksum (position 9). PeselTaggedRule {{PESEL_TAGGED}} Polish PESEL with label (e.g., PESEL: 44051401359 ). Validates checksum, preserves label prefix. PeselRule {{PESEL}} Polish PESEL numbers (11-digit personal identifiers). Validates checksum via is_valid_pesel . NipRule {{NIP}} Polish NIP numbers (plain, hyphen-separated, or wrapped in markdown). Validates checksum with weights [6,5,7,2,3,4,5,6,7] . KrsRule {{KRS}} Polish KRS numbers (10 digits, plain or hyphen-separated). Validates format only (10 digits). RegonRule {{REGON}} Polish REGON numbers (9 or 14 digits, optionally split by spaces). Validates checksum; handles both 9-digit and 14-digit forms. 2. High Certainty — Strict Format Validation # Rule Placeholder What it Detects Notes NrbRule {{NRB}} Polish NRB (bank account) numbers (26 digits, optional spaces). Supports formats: 26 digits or 2-4-4-4-4-4-4 grouping. MacAddressRule {{MAC_ADDRESS}} MAC addresses (6 octets, e.g., 00:1A:2B:3C:4D:5E ). Supports : , - , or no separators. PassportRule {{PASSPORT}} Passport numbers (2 letters + 7 digits, e.g., AB1234567 ). Case-insensitive matching. IdCardRule {{ID_CARD}} Polish ID card numbers (3 letters + 6 digits, e.g., ABC123456 ). Case-insensitive matching. SsnRule {{SSN}} US Social Security Numbers (format AAA-GG-SSSS , e.g., 123-45-6789 ). Format validation only, no checksum. 3. Medium-High Certainty — International Phone Numbers # Rule Placeholder What it Detects Notes PhoneInternationalRule {{PHONE_INTERNATIONAL}} Phone numbers with leading + and country code (e.g., +48 123 456 789 ). Matches 1-3 digit country codes with subscriber number groups. 4. Medium Certainty — Well-Structured Formats # Rule Placeholder What it Detects Notes EmailRule {{EMAIL}} E-mail addresses (e.g., user@example.com ). Permissive regex; matches local-part, @ , domain with TLD. Applied before URLs. UrlRule {{URL}} HTTP/HTTPS URLs and standalone domains (e.g., https://example.com , www.wp.pl ). Avoids code patterns like requests.post , response.json . IpRule {{IP}} , {{PORT}} IPv4, IPv6 addresses and hostname localhost . Masks ports as {{IP}}:{{PORT}} . Light octet validation; port captured separately. BankAccountRule {{BANK_ACCOUNT}} Polish IBAN (28 characters) and partially masked accounts (groups may contain X ). Exact length match; supports masked formats. 5. Medium-Low Certainty — Business Identifiers # Rule Placeholder What it Detects Notes JwtRule {{JWT}} JSON Web Tokens (3 Base64URL parts separated by dots). Validates structure; each part must be substantial (≥20 chars for header/payload). InvoiceNumberRule {{INVOICE_NUMBER}} Invoice identifiers (e.g., FV/2023/00123 , INV-2023-456 ). Case-insensitive; matches FV , INV , or INVOICE prefixes. OrderNumberRule {{ORDER_NUMBER}} E-commerce order identifiers (e.g., ORD123456 , ORDER-2023-001 ).…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/agentic-routing/claude-code/index.html","t":"Claude Code Model Swap (agenticroutingclaude_code)","s":"Routing plugins","x":"# Claude Code Model Swap (`agentic_routing_claude_code`) Model-swapping plugin for the **Anthropic-Messages-style** requests emitted by the **Claude Code** CLI. It looks at the model every request asks for, finds the Claude Code **tier** t…","h":["What the plugin changes","Quick start","1. Install the plugin package","2. Load the plugin into the router","3. Declare the models in the router","4. Point Claude Code at the router","5. Verify","How it works","Step 1 — normalize the requested name","Step 2 — match it against the configured tiers","Step 3 — rewrite and annotate","Configuration","Config file layout","Tiers and what each one replaces","Environment variables","Precedence and linting","Tuning and gotchas","Verifying the plugin","Offline dry run (no network, no router)","Router smoke test","Automated tests","Troubleshooting","Reference","Module map","Defaults at a glance","See also"],"b":"Claude Code Model Swap ( agentic_routing_claude_code ) # Model-swapping plugin for the Anthropic-Messages-style requests emitted by the Claude Code CLI. It looks at the model every request asks for, finds the Claude Code tier that owns that name, and rewrites the model to the one this router actually serves. Sonnet traffic, Opus traffic, the Haiku calls Claude Code runs in the background and the opusplan Plan Mode phase can each land on their own model, without any per-developer configuration. The plugin exists because Claude Code has no gateway-side notion of a model mapping. Pointing it at your own backend used to mean setting one environment variable per tier on every host — ANTHROPIC_DEFAULT_FABLE_MODEL , ANTHROPIC_DEFAULT_OPUS_MODEL , ANTHROPIC_DEFAULT_SONNET_MODEL , ANTHROPIC_DEFAULT_HAIKU_MODEL , CLAUDE_CODE_SUBAGENT_MODEL — and repeating it on every laptop, container image and CI runner. Here that mapping is one JSON file the router reads once. Plugin name (registry key): agentic_routing_claude_code Class: llm_router_plugins.utils.routing.agentic_routing.claude_code.plugin.ClaudeCodeRoutingPlugin Default config: llm_router_plugins/resources/routing/agentic_routing_claude_code.json Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CLAUDE_CODE_ Dependencies: none beyond the standard library — no embedding model, no FAISS What the plugin changes # apply() rewrites the payload in place and touches the model key(s) plus one annotation: Key Value model model_name of the matched tier, when the payload carried it model_name the same target, when the payload carried it routing plugin , similarity , mode , original_model , matched_model , match_type , field Only keys the payload already carries are rewritten, and only those configured in settings.model_fields are considered at all: a payload that names its model model_name never grows a model key, and a key outside model_fields keeps its value. Everything else — messages , system , tools , max_tokens , metadata — is forwarded unchanged. The plugin never rejects a request; it only ever picks a model. Fail-open, and silent about it. A payload that is not a dict, carries no configured model key, whose model no tier claims, or whose tier has no model_name , is returned as the same object it came in as — no rewrite, no log line. There is deliberately nothing to grep for on that path: an unmatched model is normal traffic, not a signal. Anything raised inside the plugin is caught, logged once as a warning, and the request still goes out unchanged. Fail-hard on configuration only. No tiers, duplicated tier names, an embedded wildcard, or one exact model name claimed by two tiers all raise at plugin construction, so the router refuses to start rather than misroute quietly. Quick start # 1. Install the plugin package # # from a checkout of this repository, into the environment the router runs in pip install -e . # this plugin needs nothing extra Unlike the semantic and Codex routing plugins, this one has no [ml] extra: it never embeds anything. 2. Load the plugin into the router # The plugin is a utils plugin: it runs inside the llm-router request pipeline and is selected by a comma-separated list of plugin identifiers. export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"agentic_routing_claude_code\" The router resolves the name against llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY , instantiates it once per process ( UtilsRegistry ) and calls apply() on every prepared request. Order matters when several utils plugins are wired together: the first one to rewrite the model wins the downstream dispatch, so put this plugin before any semantic routing plugin that might answer the same request. 3. Declare the models in the router # Every value in claude_code_modes[].model_name must be a normal, active model in the llm-router model config ( LLM_ROUTER_MODELS_CONFIG ) with a reachable provider. A name the router does not know is a hard downstream failure ( model not found ), not a plugin error. Ch…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/agentic-routing/codex/index.html","t":"Codex CLI Routing (agenticroutingcodex)","s":"Routing plugins","x":"# Codex CLI Routing (`agentic_routing_codex`) Routing plugin for the **OpenAI-Responses-style** requests emitted by the **Codex CLI** coding agent. It looks at every request that arrives with the trigger model (`auto_codex` by default), de…","h":["What the plugin changes","Quick start","1. Install the plugin package","2. Load the plugin into the router","3. Declare the models in the router","4. Point the Codex CLI at the router","5. Verify","How it works","Where it runs","Step 1 — payload normalization","Step 2 — request class","Step 3 — resolution cascade","Step 4 — keyword scoring","Step 5 — semantic similarity (optional)","Step 6 — annotating the payload","Failure behaviour","Configuration","Config file layout","Environment variables","Precedence and linting","Configuring the models","The shipped mapping","Model facts that matter here","Hard requirements on the router side","Choosing which mode gets which model","Tuning and gotchas","Verifying the plugin","Offline dry run (no network, no embedding model)","Router smoke test","Automated tests","Troubleshooting","Reference","Module map","Defaults at a glance","Reference deployment (example — your hosts will differ)","See also"],"b":"Codex CLI Routing ( agentic_routing_codex ) # Routing plugin for the OpenAI-Responses-style requests emitted by the Codex CLI coding agent. It looks at every request that arrives with the trigger model ( auto_codex by default), decides what kind of work the agent is doing right now , and rewrites payload[\"model\"] to the model configured for that work mode. Planning turns, code reviews, test runs, debugging, one-line thread titles and context compaction can then each run on the model that fits them best, while the Codex CLI keeps talking to a single stable model name. The decision cascade is deterministic first : structural signals (request class, the <collaboration_mode> block the CLI injects, keyword scoring) answer most requests offline, for free, and reproducibly. An optional embedding layer contributes cosine similarity over the mode descriptions and examples only when the cheap layers stay silent. Plugin name (registry key): agentic_routing_codex Class: llm_router_plugins.utils.routing.agentic_routing.codex.plugin.CodexRoutingPlugin Default config: llm_router_plugins/resources/routing/agentic_routing_codex.json Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_ What the plugin changes # apply() rewrites the payload in place and touches exactly three keys: Key Value model model_name of the resolved Codex work mode agent_mode name of the resolved mode ( plan , implement , test , …) routing decision metadata: plugin , similarity , source , class and ids Everything else — input , instructions , tools , reasoning , text , stream , parallel_tool_calls — is forwarded unchanged. The plugin never rejects a request; it only ever picks a model. Fail-open by design. A payload that is not a dict, does not carry the trigger model, resolves to a mode that is not configured or to a mode with an empty model_name , or that raises anywhere during parsing or classification, is returned untouched with the trigger model still in place. Fail-hard on configuration only : an inconsistent config raises at plugin construction, so the router refuses to start instead of silently misrouting every request. Quick start # 1. Install the plugin package # # from a checkout of this repository, into the environment the router runs in pip install -e . # deterministic routing only pip install -e \".[ml]\" # + CPU FAISS and sentence-transformers (semantic layer) pip install -e \".[ml-gpu]\" # + CUDA FAISS instead — never both FAISS builds at once The [ml] / [ml-gpu] extras install faiss-cpu / faiss-gpu , sentence-transformers , numpy and scipy . Without them the plugin still loads and routes deterministically — the semantic layer steps aside with a single warning. In a container image, add the package (with the extra you want) next to llm-router so both import the same sentence-transformers . 2. Load the plugin into the router # The plugin is a utils plugin: it runs inside the llm-router request pipeline and is selected by a comma-separated list of plugin identifiers. export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"agentic_routing_codex\" The router resolves the name against llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY , instantiates it once per process ( UtilsRegistry ), and calls its apply() on every prepared request, before guardrails and before a provider is chosen. Order matters if you wire several utils plugins together: the first plugin that rewrites model wins the downstream dispatch. Instantiating the plugin loads the JSON config, applies env overrides, validates them and — when the semantic layer is enabled — loads the embedding model and builds (or loads) the FAISS index. Expect a few seconds of extra startup time and a couple of hundred MB of RSS for a 300 M embedding model. 3. Declare the models in the router # Two things must exist in the llm-router model config ( LLM_ROUTER_MODELS_CONFIG , JSON): The trigger auto_codex as a declared model alias backed by a builtin provider — the CLI asks for it, the router accepts it, and the plugin re…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/semantic-biencoder/index.html","t":"Semantic BiEncoder Routing (semanticbiencoderrouting)","s":"Routing plugins","x":"# Semantic BiEncoder Routing (`semantic_biencoder_routing`) Embedding-based model selection. A set of **routing targets** — each one a name, a model and a pile of natural-language examples — is embedded once at startup into a FAISS index. …","h":["What the plugin changes","Quick start","1. Install with the ML extras","2. Load the plugin into the router","3. Make the embedding weights available","4. Declare the models in the router model config","5. Verify","How it works","Startup (constructor)","Request path","Measured behaviour","Configuration","Config file layout","Environment variables","Startup failures","Tuning and gotchas","Verifying the plugin","Live check (loads the embedding model)","Router smoke test","Automated tests","Troubleshooting","Reference","Module map","Defaults at a glance","See also"],"b":"Semantic BiEncoder Routing ( semantic_biencoder_routing ) # Embedding-based model selection. A set of routing targets — each one a name, a model and a pile of natural-language examples — is embedded once at startup into a FAISS index. Every incoming request that asks for model: \"auto\" has its last user message embedded and matched against that index by cosine similarity, and payload[\"model\"] is replaced with the model of the closest target. This is the \"teach routing by writing examples\" plugin: to change a decision, add or reword examples rather than maintain keyword lists. It needs sentence-transformers and faiss and it loads an embedding model into the router process. Plugin name (registry key): semantic_biencoder_routing Class: llm_router_plugins.utils.routing.semantic_biencoder.semantic_biencoder_routing.SemanticBiEncoderRoutingPlugin Default config: llm_router_plugins/resources/routing/semantic_biencoder.json Trigger: payload[\"model\"] equals auto after trimming ( should_route ) Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_ Dependencies: pip install -e \".[ml]\" (or [ml-gpu] ) What the plugin changes # Key Value model model_name of the winning routing target routing plugin , target_name , similarity (mean cosine of the match) Everything else in the payload is untouched. A rejected match (similarity below threshold) and a request without extractable text leave the payload as it is , so model stays \"auto\" in both cases. { \"model\" : \"qwen3.6:35b\" , \"routing\" : { \"plugin\" : \"semantic_biencoder_routing\" , \"similarity\" : 0.8224 , \"target_name\" : \"code-generation\" } } The log line that explains every decision: SemanticBiEncoderRouting: text='Napisz funkcję w Pythonie, która parsuje plik CSV' target='code-generation' similarity=0.8224 -> model=qwen3.6:35b Quick start # 1. Install with the ML extras # # from a checkout of this repository, into the environment the router runs in pip install -e \".[ml]\" # CPU FAISS + sentence-transformers pip install -e \".[ml-gpu]\" # CUDA FAISS instead — never both FAISS builds at once Without these imports this plugin does not start : the constructor raises and the router refuses to boot. 2. Load the plugin into the router # export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"semantic_biencoder_routing\" Resolved through llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY , instantiated once per process, then run on every prepared payload before guardrails and provider selection. 3. Make the embedding weights available # embedding_model is a Hugging Face id or a local directory . google/embeddinggemma-300m is gated behind the Gemma license, so either accept it and huggingface-cli login , or point at a local copy: { \"embedding_model\" : \"/models/google/embeddinggemma-300m\" } or without touching the file: export LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_MODEL = \"/models/google/embeddinggemma-300m\" A local path also removes the startup download, which matters for air-gapped or restart-heavy deployments. 4. Declare the models in the router model config # the auto alias, backed by a builtin provider (the client asks for it, this plugin replaces it); every model_name used by a routing target ( qwen3.6:35b , gpt-oss:120b , …) as a normal reachable model. // fragment of the llm-router models config \"semantic_routing\": { \"auto\": { \"providers\": [ { \"id\": \"semantic-routing__auto\", \"api_type\": \"builtin\", \"api_host\": null, \"api_token\": \"\", \"model_path\": \"\", \"input_size\": 0, \"weight\": 1.0, \"tool_calling\": false } ] } } 5. Verify # Embedding model loaded successfully. Index built: 6 targets, 79 total embeddings. SemanticBiEncoderRouting: text='…' target='code-generation' similarity=0.8224 -> model=qwen3.6:35b Startup of the shipped config (6 targets, 79 vectors) takes about 9 s on CPU including the model load. See Verifying the plugin for a runnable check. How it works # Startup (constructor) # SemanticBiEncoderConfig.from_file() — bundled JSON, or the …_CONFIG env var (raw JSON string or a path). _override_from_env…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/index.html","t":"Semantic Routing Plugins","s":"Routing plugins","x":"# Semantic Routing Plugins Three routing plugins are available for **model selection** in the **LLM‑Router** system. - `simple_semantic_routing` and `semantic_biencoder_routing` activate when `payload[\"model\"] == \"auto\"`. - `agentic_routin…","h":["Table of Contents","0. Shared Routing Layer","1. Simple Semantic Routing (Heuristic)","1.1 Architecture","1.2 Intent Categories","1.3 Model Selection Algorithm","1.4 Configuration","1.5 Environment Variable Overrides","1.6 Usage Examples","1.7 Weight Tuning Guidelines","1.8 Phrase Format Guidelines","1.9 Complexity Levels","1.10 Built-in Patterns Reference","1.11 Adding New Intents","1.12 Running Tests","2. Bi-Encoder Semantic Routing (Embedding-based)","2.1 Index Building (on first load or when the persist directory is missing)","2.2 Routing (query)","2.3 Persistence","2.4 Configuration","2.5 Usage Examples","2.6 Scoring Details","2.7 Running Tests","3. Codex Routing (Codex CLI Requests)","3.1 Module Layout","3.2 Request Classes","3.3 Detection Cascade","3.4 Similarity: Cosine over Mode Embeddings","3.5 Configuration","3.6 Environment Variable Overrides","3.7 Usage Example","3.8 Running Tests","4. Comparison: Which Plugin to Use?","Recommendation","5. File Locations"],"b":"Semantic Routing Plugins # Three routing plugins are available for model selection in the LLM‑Router system. simple_semantic_routing and semantic_biencoder_routing activate when payload[\"model\"] == \"auto\" . agentic_routing_codex activates when payload[\"model\"] equals its own trigger value ( \"auto_codex\" by default) and routes the Codex CLI's request class (main turn, title generation, compaction) and work mode. Table of Contents # 0. Shared Routing Layer 1. Simple Semantic Routing (Heuristic) 2. Bi-Encoder Semantic Routing (Embedding-based) 3. Codex Routing (Codex CLI Requests) 4. Comparison: Which Plugin to Use? 5. File Locations 0. Shared Routing Layer # semantic_biencoder_routing and agentic_routing_codex share a common layer in llm_router_plugins/utils/routing/ : Module Contents embedder.py EmbeddingRouter (BiEncoder + FAISS: chunking, index build/load/persist, route() ) and EmbeddingRouterConfig — the formal config contract the router duck-types target.py RoutingTarget — the shared target/mode dataclass ( name , model_name , description , examples ); CodexMode is a subclass of it common.py RoutingConfigBase (shared from_file / from_json protocol: ..._CONFIG env var holding raw JSON or a file path, optional default config location), env_int / env_float / env_bool / resolve_persist_dir env helpers, build_embedding_router + check_router_has_vectors , should_route (the payload[\"model\"] trigger gate) and annotate_routing (writes payload[\"model\"] + payload[\"routing\"] ) Backward compatibility: semantic_biencoder/embedder.py and semantic_biencoder/config.py re-export EmbeddingRouter / EmbeddingRouterConfig / RoutingTarget from the shared modules, so existing imports keep working. Both plugins also share the text-extraction helper llm_router_plugins/utils/text_extractor.py::extract_user_text ( messages[-1].content → user_last_statement → query → prompt → input ). 1. Simple Semantic Routing (Heuristic) # The Simple Semantic Routing plugin ( simple_semantic_routing ) performs two-stage heuristic model selection: it classifies the user's intent (code, math, creative, general) via weighted keywords, multi-word phrases, and regex patterns, then estimates input complexity (token count) to pick the most appropriate model from a configured pool. No embedding model is required — routing is a fast, pure-text classification. 1.1 Architecture # Data Flow # User Input │ ▼ classify_intent(text) ← keywords + phrases + regex patterns │ ▼ estimate_tokens(text) ← word_count × 1.25 │ ▼ complexity_level(tokens) ← simple / medium / complex │ ▼ select_model(intent, complexity) ← weighted score → model index │ ▼ Updated payload[\"model\"] Intent Classification Algorithm # Each intent is defined in simple_semantic.json with four complementary signal types : a) Keywords — single words # Each keyword has an optional weight. If no weight is specified, the default is 1.0 . \"keywords\" : [ \"code\" , \"debug\" , \"funkcja\" ], \"weights\" : { \"debug\" : 5 , \"kod\" : 4 , \"code\" : 2 } Matching \"debug\" adds 5 to the score; matching \"code\" adds 2 . b) Phrases — multi-word expressions # Phrases use the format \"text:weight\" . The default weight is 2.0 when omitted. \"phrases\" : [ \"write code:5\" , \"fix bug:4\" , \"napisz funkcję:4\" ] Matching \"write code\" adds 5 to the score. c) Regex Patterns — structural detection # Patterns add a flat +3.0 per match, useful for detecting code structures, math formulas, and question patterns. \"patterns\" : [ \"def\\\\s+\\\\w+\\\\s*\\\\(\" , \"class\\\\s+\\\\w+\\\\s*[(:]\" , \"\\\\b\\\\d+\\\\s*[+\\\\-*/^]\\\\s*\\\\d+\\\\b\" , \"\\\\bwhat\\\\s+(is|are|does)\\\\b\" ] d) Weights — keyword importance # The weights object maps individual keywords to boost factors, allowing fine-grained control: \"weights\" : { \"debug\" : 5 , // Very strong signal for code intent \"błąd\" : 5 , // Very strong signal for code intent \"kod\" : 2 , // Moderate signal \"python\" : 2 // Moderate signal } 1.2 Intent Categories # Intent Description Examples code Programming, debugging, implementation \"napisz funkcję\", \"fix bug…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/simple-semantic/index.html","t":"Simple Semantic Routing (simplesemanticrouting)","s":"Routing plugins","x":"# Simple Semantic Routing (`simple_semantic_routing`) Heuristic, two-stage model selection: it classifies the **intent** of the last user message (code / math / creative / general) with weighted keywords, phrases and regexes, estimates the…","h":["What the plugin changes","Quick start","1. Install","2. Load the plugin into the router","3. Declare the models in the router model config","4. Verify","How it works","Where it runs","Text extraction","Stage 1 — intent classification","Stage 2 — complexity estimation","Stage 3 — model selection","Configuration","Config file layout","Environment variables","Validation","Tuning and gotchas","Verifying the plugin","Offline dry run","Router smoke test","Automated tests","Troubleshooting","Reference","Defaults at a glance","Files","See also"],"b":"Simple Semantic Routing ( simple_semantic_routing ) # Heuristic, two-stage model selection: it classifies the intent of the last user message (code / math / creative / general) with weighted keywords, phrases and regexes, estimates the complexity from a rough token count, and picks a model from an ordered pool. No embedding model, no vector store, no network — the whole decision is a few string scans, so it costs microseconds and is fully reproducible. It is the entry-level routing plugin: turn model: \"auto\" into \"a sensible model for this request\" without any ML dependencies. When you outgrow keyword heuristics, move to semantic_biencoder_routing , which decides by embedding similarity instead. Plugin name (registry key): simple_semantic_routing Class: llm_router_plugins.utils.routing.simple_semantic.simple_semantic_routing.SimpleSemanticRoutingPlugin Default config: llm_router_plugins/resources/routing/simple_semantic.json Trigger: payload[\"model\"] == \"auto\" (exact match) Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_ Dependencies: none beyond the base package What the plugin changes # apply() sets exactly one key: Key Value model the pool entry selected for this intent + complexity Nothing else is added — there is no routing metadata block in the payload (unlike semantic_biencoder_routing and agentic_routing_codex ). Everything the plugin knows is written to the log: SimpleSemanticRouting: intent=code, complexity=simple (10 tokens) -> qwen3.6:35b If no user text can be found, model is set to the configured default model and a warning is logged; a payload whose model is not exactly \"auto\" is returned untouched. Quick start # 1. Install # # from a checkout of this repository, into the environment the router runs in pip install -e . No extras needed — this plugin imports only the standard library. 2. Load the plugin into the router # export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"simple_semantic_routing\" The router resolves the name in llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY , instantiates it once per process and runs apply() on every prepared payload, before guardrails and before a provider is selected. 3. Declare the models in the router model config # Everything the plugin can select must exist in the llm-router model config ( LLM_ROUTER_MODELS_CONFIG ): the auto alias itself, backed by a builtin provider — the client asks for it, the plugin replaces it; every model in the pool ( default_models values, or the LLM_ROUTER_ROUTING_SEMANTIC_MODELS list). // fragment of the llm-router models config \"semantic_routing\": { \"auto\": { \"providers\": [ { \"id\": \"semantic-routing__auto\", \"api_type\": \"builtin\", \"api_host\": null, \"api_token\": \"\", \"model_path\": \"\", \"input_size\": 0, \"weight\": 1.0, \"tool_calling\": false } ] } } A pool entry the router does not know is a hard downstream error, not a routing no-op. 4. Verify # Send a request with \"model\": \"auto\" and look for the log line, or run the verification snippet . Two lines to remember: SimpleSemanticRouting: intent=code, complexity=simple (10 tokens) -> qwen3.6:35b No text content found, using default model gpt-oss:120b How it works # payload ──▶ extract text ──▶ stage 1: intent ──▶ stage 2: complexity ──▶ stage 3: pool index ──▶ payload[\"model\"] Where it runs # Inside the llm-router utils pipeline. The plugin is stateless: no cache, no session, no learned state, so every replica answers identically. Text extraction # Text is located by the shared helper llm_router_plugins.utils.text_extractor.extract_user_text , in this priority: payload[\"messages\"][-1][\"content\"] — the last message of a chat history (earlier turns are not seen) payload[\"user_last_statement\"] payload[\"query\"] payload[\"prompt\"] payload[\"input\"] Only that one string is scored, so intent detection always reflects the newest user turn. Stage 1 — intent classification # Every intent in intents accumulates a score from three signal kinds; the highest score above zero wins, ties go to the intent declared first in …","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/rag/langchain-rag.html","t":"LangChain RAG Plugin for LLM‑Router","s":"Utility plugins","x":"# LangChain RAG Plugin for LLM‑Router A lightweight **Retrieval‑Augmented Generation (RAG)** plugin that lets you attach any local knowledge base to the LLM‑Router without touching the original application code. It builds the knowledge bas…","h":["Table of Contents","Features","Installation","CLI – Indexing & Searching","Searching","Example workflow","Error handling","License & Credits","Happy augmenting! 🚀"],"b":"LangChain RAG Plugin for LLM‑Router # A lightweight Retrieval‑Augmented Generation (RAG) plugin that lets you attach any local knowledge base to the LLM‑Router without touching the original application code. It builds the knowledge base with LangChain , stores vectors in FAISS (fallback to Milvus is possible), and injects the retrieved context into every LLM request. Table of Contents # Features Installation Configuration (environment variables) CLI – Indexing & Searching Integrating the plugin into LLM‑Router Example workflow Error handling License & Credits Features # Feature Description Indexing Recursively reads a directory of .md , .txt , .html (or any extensions you pass) → splits documents into overlapping token windows → embeds chunks with a transformer model → stores them in a persistent FAISS index. Searching Given a user query, retrieves the top‑N most similar chunks (default 10) using cosine similarity. Context injection Retrieved chunks are appended (with a clear prefix) to the user’s last message, so the downstream LLM can use the extra knowledge automatically. Full configurability All hyper‑parameters (collection name, embedder model, device, chunk size, overlap, persistence directory) are set via environment variables. Fail‑safe If the RAG subsystem is disabled or mis‑configured, the plugin raises a clear exception while the rest of the router keeps working. CLI helpers llm-router-rag-langchain command provides convenient index and search sub‑commands, plus an interactive REPL mode. Installation # Only the optional RAG dependencies are required; they are not installed with the core LLM‑Router. ```shell script Activate your virtualenv / conda env first # pip install faiss-cpu tqdm langchain langchain-community transformers torch > **Note** – If you prefer GPU‑accelerated FAISS, replace `faiss-cpu` with `faiss-gpu`. --- ## Configuration (environment variables) | Variable | Default | Description | |------------------------------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `LLM_ROUTER_LANGCHAIN_RAG_COLLECTION` | *must be set* | Name of the FAISS collection (e.g. `sample_collection`). | | `LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER` | *must be set* | Hugging Face model identifier for the embedder (e.g. `sentence-transformers/all-MiniLM-L6-v2`). | | `LLM_ROUTER_LANGCHAIN_RAG_DEVICE` | `cpu` | Torch device (`cpu`, `cuda:0`, `cuda:1`, …). | | `LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE` | `1024` | Number of tokens per chunk. | | `LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP` | `100` | Overlap in tokens between consecutive chunks. | | `LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR` | *none* | Directory where the FAISS index and docstore are persisted. If omitted, a folder named after the collection is created under `./workdir/plugins/utils/rag/langchain/`. | ### Example export block (add to your shell profile or a `.env` file) ```shell script export LLM_ROUTER_LANGCHAIN_RAG_COLLECTION=\"${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION:-sample_collection}\" export LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER=\"${LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER:-sentence-transformers/all-MiniLM-L6-v2}\" export LLM_ROUTER_LANGCHAIN_RAG_DEVICE=\"${LLM_ROUTER_LANGCHAIN_RAG_DEVICE:-cpu}\" export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE=\"${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE:-200}\" export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP=\"${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP:-50}\" export LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR=\"${LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR:-./workdir/plugins/utils/rag/langchain/${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION}}\" CLI – Indexing & Searching # The plugin ships with a convenient entry‑point: ```shell script llm-router-rag-langchain [options] ### Indexing ```shell script # Index Markdown, plain‑text and HTML files from a repository llm-router-rag-langchain index --path \"../llm-router\" --ext .md .txt .html You can run …","r":"llm-router-plugins"}]}