{"version":"0.1.4","pages":[{"k":"index-2.html","t":"Plugins overview","s":"Plugins overview","x":"## Overview The **LLM‑Router** project ships with a modular plugin system that lets you plug‑in **anonymizers** (also called *maskers*) and **guardrails** into request‑processing pipelines. Each plugin implements a tiny, well‑defined inter…","h":["Overview","1. Anonymizers (Maskers)","1.1 What they do","1.2 Built‑in anonymizer plugins","1.3 How a masker is used","2. Guardrails","2.1 What they do","2.2 Built‑in guardrail plugins","2.3 How a guardrail is used","2.5 ML-Based PII Classification","How it complements regex maskers","Quick start","2.6 Polish Identification Regex Patterns","Available Polish rules","How they work","Adding rules to the pipeline","2.7 Semantic Routing (Model Selection)","2.7.1 Simple Semantic Routing (Heuristic)","2.7.2 Bi-Encoder Semantic Routing (Model Selection)","2.7.3 Codex CLI routing (auto_codex)","2.7.4 Claude Code Model Swap (claude-* → configured models)","2.7.5 Agentic knowledge layer (agentic_knowledge)","3. Pipelines","3.1 Registration","3.2 Configuration","4. Adding a New Plugin","5. Retrieval‑Augmented Generation (RAG) Support","5.1 What the plugin does","5.2 Environment variables"],"b":"Overview # The LLM‑Router project ships with a modular plugin system that lets you plug‑in anonymizers (also called maskers ) and guardrails into request‑processing pipelines. Each plugin implements a tiny, well‑defined interface ( apply ) and can be composed in an ordered list to form a * pipeline *. Pipelines are instantiated by the MaskerPipeline and GuardrailPipeline classes and are driven automatically by the endpoint logic in endpoint_i.py . 1. Anonymizers (Maskers) # 1.1 What they do # Goal – Remove or replace personally‑identifiable information (PII) from a payload before it reaches the LLM or an external service. Typical strategy – Run a pipeline of maskers that locate spans corresponding to IDs, emails, IPs, etc., and replace each span with a placeholder such as {{MASKED_ITEM}} . 1.2 Built‑in anonymizer plugins # Plugin Description Technical notes FastMaskerPlugin ( fast_masker_plugin.py ) Thin wrapper around the FastMasker utility class. Receives a JSON‑compatible payload and returns the same payload with all detected PII masked. Implements PluginInterface . The heavy lifting is delegated to FastMasker.mask_payload(payload) . No extra I/O; the FastMasker instance is created once in __init__ . 1.3 How a masker is used # The endpoint (e.g. EndpointI._do_masking_if_needed ) checks the global flag FORCE_MASKING . If enabled, it creates a MaskerPipeline with the list of masker plugin identifiers (e.g. [\"fast_masker\"] ). The pipeline calls each plugin’s apply method sequentially, feeding the output of one as the input of the next. The final payload – now stripped of PII – proceeds to the rest of the request flow (guardrails, model dispatch, etc.). 2. Guardrails # 2.1 What they do # Goal – Verify that a request (or its response) complies with policy rules (e.g. no hateful, illegal, or unsafe content). Typical strategy – Split the payload into manageable text chunks, run a pipeline of guardrails, aggregate per‑chunk scores, and decide whether the overall request is safe. 2.2 Built‑in guardrail plugins # Plugin Description Technical notes NASKGuardPlugin ( nask_guard_plugin.py ) HTTP‑based guardrail that forwards the payload to the external NASK guardrail service ( /nask_guard endpoint) and returns a boolean safe flag together with the raw response. Inherits from HttpPluginInterface . The apply method calls _request(payload) (provided by the base class) and extracts results[\"safe\"] . Errors are caught and logged; on failure the plugin returns (False, {}) . SojkaGuardPlugin ( sojka_guard_plugin.py ) HTTP‑based guardrail that forwards the payload to the Sójka guardrail service ( /sojka_guard endpoint) and returns a safety flag. Mirrors the design of NASKGuardPlugin . The endpoint_url is built from the LLM_ROUTER_GUARDRAIL_SOJKA_GUARD_HOST environment variable. On success it returns (True, response) , otherwise (False, {}) . (Implicit) GuardrailProcessor ( processor.py ) Core logic used by the internal NASK guardrail Flask route ( nask_guardrail ). Tokenises the payload, creates overlapping chunks, runs a Hugging‑Face text‑classification pipeline, and produces a detailed safety report. Handles model loading ( AutoTokenizer , pipeline(\"text‑classification\") ), chunking ( _chunk_text ), and scoring thresholds ( MIN_SCORE_FOR_SAFE , MIN_SCORE_FOR_NOT_SAFE ). Returns a dict: {\"safe\": &lt;bool&gt;, \"detailed\": [...]} . 2.3 How a guardrail is used # The endpoint calls _is_request_guardrail_safe(payload) (or the analogous response guardrail). If FORCE_GUARDRAIL_REQUEST is true, a GuardrailPipeline is built from the configured plugin IDs (e.g. [\"nask_guard\", \"sojka_guard\"] ). The pipeline iterates over each guardrail plugin; each apply returns (is_safe, message) . The first plugin that reports is_safe=False short‑circuits the pipeline and the request is rejected with a 400/500 error payload. 2.5 ML-Based PII Classification # For cases where regex patterns alone are insufficient (e.g. context-dependent PII detection), the project integ…","r":"llm-router-plugins"},{"k":"llm-router-plugins/maskers/fast-masker/index.html","t":"Overview","s":"Masker plugins","x":"## Overview The **fast_masker** plugin provides a simple, rule‑based engine that scans a piece of text and replaces sensitive data ( e‑mail addresses, IPs, URLs, phone numbers, Polish PESEL identifiers, etc.) with clearly marked placeholde…","h":["Overview","Masking Rules","1. Highest Certainty — Checksum Validated Identifiers","2. High Certainty — Strict Format Validation","3. Medium-High Certainty — International Phone Numbers","4. Medium Certainty — Well-Structured Formats","5. Medium-Low Certainty — Business Identifiers","6. Lower Certainty — Pattern-Based with Context","7. Format-Based — Specific Patterns","8. Lowest Certainty — Generic Patterns","9. Beta Features","Disabled Rules (Too Noisy)","Utility Validators","Polish Identification Numbers","Financial Numbers","Vehicle and Transport","International Identification","Network and Security","Business Identifiers"],"b":"Overview # The fast_masker plugin provides a simple, rule‑based engine that scans a piece of text and replaces sensitive data ( e‑mail addresses, IPs, URLs, phone numbers, Polish PESEL identifiers, etc.) with clearly marked placeholders. The core component is the :class: ~llm_router_plugins.plugins.fast_masker.core.masker.FastMasker , which receives an ordered list of rule objects and applies each rule sequentially to the input text. Because the rules are applied in the order they are supplied, you can control precedence (e.g., replace URLs before e‑mails if needed). Masking Rules # Rules are applied in order from highest certainty (checksum-validated) to lowest certainty (pattern-based). This ordering minimizes false positives and ensures the most reliable identifiers are masked first. 1. Highest Certainty — Checksum Validated Identifiers # Rule Placeholder What it Detects Notes CreditCardRule {{CREDIT_CARD}} Credit card numbers (13-19 digits with optional spaces/dashes, e.g., 4532 1234 5678 9010 ). Validates using Luhn algorithm checksum. VinRule {{VIN}} Vehicle Identification Numbers (17 characters, e.g., 1HGBH41JXMN109186 ). Validates using ISO 3779 checksum (position 9). PeselTaggedRule {{PESEL_TAGGED}} Polish PESEL with label (e.g., PESEL: 44051401359 ). Validates checksum, preserves label prefix. PeselRule {{PESEL}} Polish PESEL numbers (11-digit personal identifiers). Validates checksum via is_valid_pesel . NipRule {{NIP}} Polish NIP numbers (plain, hyphen-separated, or wrapped in markdown). Validates checksum with weights [6,5,7,2,3,4,5,6,7] . KrsRule {{KRS}} Polish KRS numbers (10 digits, plain or hyphen-separated). Validates format only (10 digits). RegonRule {{REGON}} Polish REGON numbers (9 or 14 digits, optionally split by spaces). Validates checksum; handles both 9-digit and 14-digit forms. 2. High Certainty — Strict Format Validation # Rule Placeholder What it Detects Notes NrbRule {{NRB}} Polish NRB (bank account) numbers (26 digits, optional spaces). Supports formats: 26 digits or 2-4-4-4-4-4-4 grouping. MacAddressRule {{MAC_ADDRESS}} MAC addresses (6 octets, e.g., 00:1A:2B:3C:4D:5E ). Supports : , - , or no separators. PassportRule {{PASSPORT}} Passport numbers (2 letters + 7 digits, e.g., AB1234567 ). Case-insensitive matching. IdCardRule {{ID_CARD}} Polish ID card numbers (3 letters + 6 digits, e.g., ABC123456 ). Case-insensitive matching. SsnRule {{SSN}} US Social Security Numbers (format AAA-GG-SSSS , e.g., 123-45-6789 ). Format validation only, no checksum. 3. Medium-High Certainty — International Phone Numbers # Rule Placeholder What it Detects Notes PhoneInternationalRule {{PHONE_INTERNATIONAL}} Phone numbers with leading + and country code (e.g., +48 123 456 789 ). Matches 1-3 digit country codes with subscriber number groups. 4. Medium Certainty — Well-Structured Formats # Rule Placeholder What it Detects Notes EmailRule {{EMAIL}} E-mail addresses (e.g., user@example.com ). Permissive regex; matches local-part, @ , domain with TLD. Applied before URLs. UrlRule {{URL}} HTTP/HTTPS URLs and standalone domains (e.g., https://example.com , www.wp.pl ). Avoids code patterns like requests.post , response.json . IpRule {{IP}} , {{PORT}} IPv4, IPv6 addresses and hostname localhost . Masks ports as {{IP}}:{{PORT}} . Light octet validation; port captured separately. BankAccountRule {{BANK_ACCOUNT}} Polish IBAN (28 characters) and partially masked accounts (groups may contain X ). Exact length match; supports masked formats. 5. Medium-Low Certainty — Business Identifiers # Rule Placeholder What it Detects Notes JwtRule {{JWT}} JSON Web Tokens (3 Base64URL parts separated by dots). Validates structure; each part must be substantial (≥20 chars for header/payload). InvoiceNumberRule {{INVOICE_NUMBER}} Invoice identifiers (e.g., FV/2023/00123 , INV-2023-456 ). Case-insensitive; matches FV , INV , or INVOICE prefixes. OrderNumberRule {{ORDER_NUMBER}} E-commerce order identifiers (e.g., ORD123456 , ORDER-2023-001 ).…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/agentic-routing/claude-code/index.html","t":"Claude Code Model Swap (agenticroutingclaude_code)","s":"Routing plugins","x":"# Claude Code Model Swap (`agentic_routing_claude_code`) Model-swapping plugin for the **Anthropic-Messages-style** requests emitted by the **Claude Code** CLI. It looks at the model every request asks for, finds the Claude Code **tier** t…","h":["What the plugin changes","Quick start","1. Install the plugin package","2. Load the plugin into the router","3. Declare the models in the router","4. Point Claude Code at the router","5. Verify","How it works","Step 1 — normalize the requested name","Step 2 — match it against the configured tiers","Step 3 — rewrite and annotate","Configuration","Config file layout","Tiers and what each one replaces","Environment variables","Precedence and linting","Tuning and gotchas","Verifying the plugin","Offline dry run (no network, no router)","Router smoke test","Automated tests","Troubleshooting","Reference","Module map","Defaults at a glance","See also"],"b":"Claude Code Model Swap ( agentic_routing_claude_code ) # Model-swapping plugin for the Anthropic-Messages-style requests emitted by the Claude Code CLI. It looks at the model every request asks for, finds the Claude Code tier that owns that name, and rewrites the model to the one this router actually serves. Sonnet traffic, Opus traffic, the Haiku calls Claude Code runs in the background and the opusplan Plan Mode phase can each land on their own model, without any per-developer configuration. The plugin exists because Claude Code has no gateway-side notion of a model mapping. Pointing it at your own backend used to mean setting one environment variable per tier on every host — ANTHROPIC_DEFAULT_FABLE_MODEL , ANTHROPIC_DEFAULT_OPUS_MODEL , ANTHROPIC_DEFAULT_SONNET_MODEL , ANTHROPIC_DEFAULT_HAIKU_MODEL , CLAUDE_CODE_SUBAGENT_MODEL — and repeating it on every laptop, container image and CI runner. Here that mapping is one JSON file the router reads once. Plugin name (registry key): agentic_routing_claude_code Class: llm_router_plugins.utils.routing.agentic_routing.claude_code.plugin.ClaudeCodeRoutingPlugin Default config: llm_router_plugins/resources/routing/agentic_routing_claude_code.json Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CLAUDE_CODE_ Dependencies: none beyond the standard library — no embedding model, no FAISS What the plugin changes # apply() rewrites the payload in place and touches the model key(s) plus one annotation: Key Value model model_name of the matched tier, when the payload carried it model_name the same target, when the payload carried it routing plugin , similarity , mode , original_model , matched_model , match_type , field Only keys the payload already carries are rewritten, and only those configured in settings.model_fields are considered at all: a payload that names its model model_name never grows a model key, and a key outside model_fields keeps its value. Everything else — messages , system , tools , max_tokens , metadata — is forwarded unchanged. The plugin never rejects a request; it only ever picks a model. Fail-open, and silent about it. A payload that is not a dict, carries no configured model key, whose model no tier claims, or whose tier has no model_name , is returned as the same object it came in as — no rewrite, no log line. There is deliberately nothing to grep for on that path: an unmatched model is normal traffic, not a signal. Anything raised inside the plugin is caught, logged once as a warning, and the request still goes out unchanged. Fail-hard on configuration only. No tiers, duplicated tier names, an embedded wildcard, or one exact model name claimed by two tiers all raise at plugin construction, so the router refuses to start rather than misroute quietly. Quick start # 1. Install the plugin package # # from a checkout of this repository, into the environment the router runs in pip install -e . # this plugin needs nothing extra Unlike the semantic and Codex routing plugins, this one has no [ml] extra: it never embeds anything. 2. Load the plugin into the router # The plugin is a utils plugin: it runs inside the llm-router request pipeline and is selected by a comma-separated list of plugin identifiers. export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"agentic_routing_claude_code\" The router resolves the name against llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY , instantiates it once per process ( UtilsRegistry ) and calls apply() on every prepared request. Order matters when several utils plugins are wired together: the first one to rewrite the model wins the downstream dispatch, so put this plugin before any semantic routing plugin that might answer the same request. 3. Declare the models in the router # Every value in claude_code_modes[].model_name must be a normal, active model in the llm-router model config ( LLM_ROUTER_MODELS_CONFIG ) with a reachable provider. A name the router does not know is a hard downstream failure ( model not found ), not a plugin error. Ch…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/agentic-routing/codex/index.html","t":"Codex CLI Routing (agenticroutingcodex)","s":"Routing plugins","x":"# Codex CLI Routing (`agentic_routing_codex`) Routing plugin for the **OpenAI-Responses-style** requests emitted by the **Codex CLI** coding agent. It looks at every request that arrives with the trigger model (`auto_codex` by default), de…","h":["Calibration and evaluation guides (PL / EN)","What the plugin changes","Quick start","1. Install the plugin package","2. Load the plugin into the router","3. Declare the models in the router","4. Point the Codex CLI at the router","5. Verify","How it works","Where it runs","Step 1 — payload normalization","Step 2 — request class","Step 3 — resolution cascade","Step 4 — keyword scoring","Step 5 — semantic similarity (optional)","Step 6 — annotating the payload","Failure behaviour","Configuration","Config file layout","Environment variables","Session memory (shared, optional)","Precedence and linting","Configuring the models","The shipped mapping","Model facts that matter here","Hard requirements on the router side","Choosing which mode gets which model","Tuning and gotchas","Verifying the plugin","Offline dry run (no network, no embedding model)","Router smoke test","Automated tests","Semantic quality evaluation and calibration","Troubleshooting","Reference","Module map","Defaults at a glance","Reference deployment (example — your hosts will differ)","See also"],"b":"Codex CLI Routing ( agentic_routing_codex ) # Routing plugin for the OpenAI-Responses-style requests emitted by the Codex CLI coding agent. It looks at every request that arrives with the trigger model ( auto_codex by default), decides what kind of work the agent is doing right now , and rewrites payload[\"model\"] to the model configured for that work mode. Planning turns, code reviews, test runs, debugging, one-line thread titles and context compaction can then each run on the model that fits them best, while the Codex CLI keeps talking to a single stable model name. The decision cascade is deterministic first : structural signals (request class, the &lt;collaboration_mode&gt; block the CLI injects, keyword scoring) answer most requests offline, for free, and reproducibly. An optional embedding layer contributes cosine similarity over the mode descriptions and examples only when the cheap layers stay silent. Plugin name (registry key): agentic_routing_codex Class: llm_router_plugins.utils.routing.agentic_routing.codex.plugin.CodexRoutingPlugin Default config: llm_router_plugins/resources/routing/agentic_routing_codex.json Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_ Calibration and evaluation guides (PL / EN) # The full guides are stored alongside this README in llm_router_plugins/utils/routing/agentic_routing/codex/ : CODEX_EVAL_HOWTO_PL.md — Polski CODEX_EVAL_HOWTO_EN.md — English They cover configuration files and ENV overrides, heuristic scoring versus embedding similarity, descriptions/examples and threshold/margin calibration, session-separated calibration/holdout datasets, the evaluation CLI, report variants, precision/recall and mode-transition metrics, and safe deployment with index rebuilding. The goal is better work-mode selection , not tuning for particular response-model names. For automated threshold/margin tuning and evaluation, see scripts/codex-tune-eval.sh in the repository root and section 6 of either guide. The script selects candidates on calibration, then compares source and selected configurations on holdout; it does not overwrite production configuration or use production Redis. Use --no-semantic for deterministic evaluation without tuning. What the plugin changes # apply() rewrites the payload in place and touches exactly three keys: Key Value model model_name of the resolved Codex work mode agent_mode name of the resolved mode ( plan , implement , test , …) routing decision metadata: plugin , similarity , source , class and ids Everything else — input , instructions , tools , reasoning , text , stream , parallel_tool_calls — is forwarded unchanged. The plugin never rejects a request; it only ever picks a model. Fail-open by design. A payload that is not a dict, does not carry the trigger model, resolves to a mode that is not configured or to a mode with an empty model_name , or that raises anywhere during parsing or classification, is returned untouched with the trigger model still in place. Fail-hard on required routing configuration : invalid triggers, modes or routing rules reject construction. Optional memory is fail-open , including invalid memory policy/ENV, a missing Redis host or client library, and connection/read/write failures: routing remains stateless. Quick start # 1. Install the plugin package # # from a checkout of this repository, into the environment the router runs in pip install -e . # deterministic routing only pip install -e \".[ml]\" # + CPU FAISS and sentence-transformers (semantic layer) pip install -e \".[ml-gpu]\" # + CUDA FAISS instead — never both FAISS builds at once The [ml] / [ml-gpu] extras install faiss-cpu / faiss-gpu , sentence-transformers , numpy and scipy . Without them the plugin still loads and routes deterministically — the semantic layer steps aside with a single warning. In a container image, add the package (with the extra you want) next to llm-router so both import the same sentence-transformers . 2. Load the plugin into the router # The plugin is a…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/agentic-routing/codex/codex-eval-howto-en.html","t":"Codex routing calibration and evaluation","s":"Routing plugins","x":"# Codex routing calibration and evaluation Wersja polska: [CODEX_EVAL_HOWTO_PL.md](CODEX_EVAL_HOWTO_PL.md). ## 1. What you are actually calibrating This guide primarily covers `agentic_routing_codex` and the implementation in this reposito…","h":["1. What you are actually calibrating","2. What the configuration files and settings are for","Key Codex configuration fields","3. Understanding scores and similarity","Heuristic scoring","Embedding scores","4. What actually goes into embeddings","5. Preparing a good calibration dataset","6. Running evaluate","Automated tuning and evaluation","A. Reference run without embeddings","B. Embedding evaluation on calibration","C. Final holdout evaluation","Important evaluator behavior","7. Reading report variants","Key metrics","8. A practical embedding calibration procedure","Step 1: freeze a reference point","Step 2: identify where errors occur","Step 3: improve descriptions and examples","Step 4: select threshold and margin","Step 5: tune other parameters only afterward","Step 6: select configuration by quality, not acceptance count","9. Calibrating heuristics","10. Confirming results and deploying configuration","Deployment","Persistent index","11. What the current report tells you","12. Short workflow checklist"],"b":"Codex routing calibration and evaluation # Wersja polska: CODEX_EVAL_HOWTO_PL.md . 1. What you are actually calibrating # This guide primarily covers agentic_routing_codex and the implementation in this repository described during the October 6–7, 2026 verification. The values and results below are reference points from that review, not a guarantee that they remain current after subsequent configuration or code changes. The goal is to identify the current work mode correctly, and only then select the model assigned to it. You are not tuning routing around the name of a particular response-generating model. There are three independent mechanisms: Structural rules and work phase — an explicit mode, Plan Mode, titles, compaction, tool calls, and their results. Heuristics — scoring keywords, phrases, and regular expressions. Semantics — embedding similarity between the current activity and mode descriptions and examples. Optional phase memory in Redis complements these mechanisms. Resolution order: explicit mode → request class: title / compaction → Plan Mode declaration → current phase / trustworthy phase memory → heuristics → embeddings → fallback → mode-to-model mapping Important: if heuristics already make an incorrect decision, lowering the embedding threshold will not fix it. The semantic layer is not consulted in that case. Calibration in the current system does not mean training or fine-tuning the embedding model . It means selecting an embedding model, descriptions, examples, indexing parameters, and acceptance thresholds. 2. What the configuration files and settings are for # File / setting Purpose llm_router_plugins/resources/routing/agentic_routing_codex.json Codex modes, rules, semantic examples, thresholds, and mode-to-model mappings. tests/data/codex_routing_quality.json Requests with expected modes. Used for evaluation; not automatically added to the embedding index. tests/data/codex_routing_baseline.json Frozen earlier predictions for comparisons on common cases. codex-routing-holdout-verification.json Saved output from a particular replay. A report, not configuration. LLM_ROUTER_MODELS_CONFIG Model configuration in the router application: availability, provider, serving parameters, etc. Does not define examples for work-mode classification. simple_semantic.json Configuration for a separate heuristic plugin. Despite its name, it does not require embeddings. semantic_biencoder.json Configuration for a separate embedding-based general routing plugin. Does not replace the Codex configuration. agentic_routing_claude_code.json Configuration for a different plugin, for Claude Code. Changing semantic_biencoder.json does not automatically calibrate Codex . The embedding engine is shared, but the configurations and decision layers are separate. Key Codex configuration fields # Field Meaning embedding_model The model that creates vectors; a local path or model identifier. Not the model answering the user. settings.trigger_model The alias that activates the plugin, auto_codex by default. settings.fallback_mode Mode selected when no mechanism resolves the request, currently implement . codex_modes[].name Mode name, such as review . codex_modes[].model_name Model used after this mode is selected. codex_modes[].description Description of the mode's meaning, indexed semantically. codex_modes[].examples Examples of activities belonging to the mode, indexed semantically. keywords , phrases , patterns , weights Heuristic signals, not embedding examples. settings.phase Activity interpretation rules: commands, tools, test paths, announcements, etc. settings.semantic Indexing and semantic acceptance parameters. settings.vector_store_path Optional directory for a persistent semantic index. Not session memory. Redis is a separate mechanism. Configure its connection using independent environment variables, not by adding connection details to the embedding configuration or automatically reusing Redis settings for auth. 3. Understanding s…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/agentic-routing/codex/codex-eval-howto-pl.html","t":"Kalibracja i ewaluacja routingu Codexa","s":"Routing plugins","x":"# Kalibracja i ewaluacja routingu Codexa English version: [CODEX_EVAL_HOWTO_EN.md](CODEX_EVAL_HOWTO_EN.md). ## 1. Co właściwie kalibrujesz Ta instrukcja dotyczy przede wszystkim `agentic_routing_codex` i implementacji w tym repozytorium op…","h":["1. Co właściwie kalibrujesz","2. Do czego służą poszczególne konfiguracje i pliki","Najważniejsze pola konfiguracji Codexa","3. Jak rozumieć score i similarity","Punktacja heurystyczna","Wynik embeddingowy","4. Co rzeczywiście trafia do embeddingów","5. Jak przygotować dobry zbiór do kalibracji","6. Jak uruchamiać evaluate","Automatyczne strojenie i eval","A. Punkt odniesienia bez embeddingów","B. Pomiar embeddingów na kalibracji","C. Końcowy pomiar na holdout","Ważne zachowania evaluate","7. Jak czytać warianty raportu","Najważniejsze metryki","8. Praktyczna procedura kalibracji embeddingów","Krok 1: zamroź punkt odniesienia","Krok 2: sprawdź, gdzie powstały błędy","Krok 3: poprawiaj opisy i przykłady","Krok 4: dobierz threshold i margin","Krok 5: dopiero potem strojenie pozostałych parametrów","Krok 6: wybierz konfigurację według jakości, nie liczby akceptacji","9. Jak kalibrować heurystyki","10. Jak potwierdzić wynik i wdrożyć konfigurację","Wdrożenie","Trwały indeks","11. Co mówi obecny raport","12. Skrócona kolejność pracy"],"b":"Kalibracja i ewaluacja routingu Codexa # English version: CODEX_EVAL_HOWTO_EN.md . 1. Co właściwie kalibrujesz # Ta instrukcja dotyczy przede wszystkim agentic_routing_codex i implementacji w tym repozytorium opisanej podczas weryfikacji z 6–7 października 2026 r. Wartości i wyniki przytoczone poniżej są punktem odniesienia z tego przeglądu, nie gwarancją aktualności po kolejnych zmianach konfiguracji lub kodu. Cel: poprawnie rozpoznać bieżący tryb pracy, a dopiero potem wybrać przypisany do niego model. Nie stroisz routingu pod nazwę konkretnego modelu generującego odpowiedź. Są trzy niezależne mechanizmy: Reguły strukturalne i faza pracy — jawny tryb, Plan Mode, tytuł, kompaktowanie, wykonane narzędzia i ich wyniki. Heurystyki — punktacja słów, fraz i wyrażeń regularnych. Semantyka — podobieństwo embeddingów aktualnej czynności do opisów i przykładów trybów. Do tego dochodzi opcjonalna pamięć fazy w Redisie. Kolejność rozstrzygania: jawny tryb → klasa żądania: tytuł / kompaktowanie → deklaracja Plan Mode → aktualna faza / wiarygodna pamięć fazy → heurystyki → embeddingi → fallback → mapowanie trybu na model Ważne: jeśli zła decyzja zapadnie już w heurystykach, obniżenie progu embeddingów jej nie naprawi. Semantyka nie jest wtedy pytana. Kalibracja w obecnym systemie nie oznacza trenowania ani fine-tuningu modelu embeddingowego . Oznacza dobór modelu embeddingowego, opisów, przykładów, parametrów indeksowania i progów akceptacji. 2. Do czego służą poszczególne konfiguracje i pliki # Plik / ustawienie Rola llm_router_plugins/resources/routing/agentic_routing_codex.json Tryby Codexa, reguły, przykłady semantyczne, progi i mapowanie trybów na modele. tests/data/codex_routing_quality.json Zbiór żądań z oczekiwanymi trybami. Służy do pomiaru, nie jest automatycznie dodawany do indeksu embeddingów. tests/data/codex_routing_baseline.json Zamrożone wcześniejsze predykcje do porównań na wspólnych przypadkach. codex-routing-holdout-verification.json Zapisany wynik konkretnego replayu. To raport, nie konfiguracja. LLM_ROUTER_MODELS_CONFIG Konfiguracja modeli po stronie aplikacji routera: dostępność, dostawca, parametry obsługi itd. Nie definiuje przykładów do klasyfikowania trybów. simple_semantic.json Konfiguracja osobnego pluginu heurystycznego. Mimo nazwy nie wymaga embeddingów. semantic_biencoder.json Konfiguracja osobnego pluginu embeddingowego do ogólnego routingu. Nie zastępuje konfiguracji Codexa. agentic_routing_claude_code.json Konfiguracja innego pluginu, dla Claude Code. Zmiana semantic_biencoder.json nie kalibruje automatycznie Codexa . Współdzielony jest silnik embeddingowy, ale konfiguracje i warstwy decyzyjne są osobne. Najważniejsze pola konfiguracji Codexa # Pole Znaczenie embedding_model Model tworzący wektory; ścieżka lokalna lub identyfikator modelu. To nie model odpowiadający użytkownikowi. settings.trigger_model Alias uruchamiający plugin, domyślnie auto_codex . settings.fallback_mode Tryb wybierany, gdy żaden mechanizm nie rozstrzygnie, obecnie implement . codex_modes[].name Nazwa trybu, np. review . codex_modes[].model_name Model używany po wybraniu tego trybu. codex_modes[].description Opis znaczenia trybu, indeksowany semantycznie. codex_modes[].examples Przykłady czynności należących do trybu, indeksowane semantycznie. keywords , phrases , patterns , weights Sygnały dla heurystyk, a nie przykłady embeddingowe. settings.phase Reguły interpretacji czynności: komendy, narzędzia, ścieżki testów, zapowiedzi itd. settings.semantic Parametry indeksowania i akceptacji odpowiedzi semantycznej. settings.vector_store_path Opcjonalny katalog trwałego indeksu semantycznego. To nie pamięć sesji. Redis jest osobnym mechanizmem. Jego połączenie konfigurujesz niezależnymi zmiennymi ENV , nie dopisujesz danych połączenia do konfiguracji embeddingów ani nie korzystasz automatycznie z ustawień Redis dla auth. 3. Jak rozumieć score i similarity # Punktacja heurystyczna # Aktualne domyślne wagi: keyword = 1 phrase = 2 pattern = 3…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/semantic-biencoder/index.html","t":"Semantic BiEncoder Routing (semanticbiencoderrouting)","s":"Routing plugins","x":"# Semantic BiEncoder Routing (`semantic_biencoder_routing`) Embedding-based model selection. A set of **routing targets** — each one a name, a model and a pile of natural-language examples — is embedded once at startup into a FAISS index. …","h":["What the plugin changes","Quick start","1. Install with the ML extras","2. Load the plugin into the router","3. Make the embedding weights available","4. Declare the models in the router model config","5. Verify","How it works","Startup (constructor)","Request path","Measured behaviour","Configuration","Config file layout","Environment variables","Startup failures","Tuning and gotchas","Verifying the plugin","Live check (loads the embedding model)","Router smoke test","Automated tests","Troubleshooting","Reference","Module map","Defaults at a glance","See also"],"b":"Semantic BiEncoder Routing ( semantic_biencoder_routing ) # Embedding-based model selection. A set of routing targets — each one a name, a model and a pile of natural-language examples — is embedded once at startup into a FAISS index. Every incoming request that asks for model: \"auto\" has its last user message embedded and matched against that index by cosine similarity, and payload[\"model\"] is replaced with the model of the closest target. This is the \"teach routing by writing examples\" plugin: to change a decision, add or reword examples rather than maintain keyword lists. It needs sentence-transformers and faiss and it loads an embedding model into the router process. Plugin name (registry key): semantic_biencoder_routing Class: llm_router_plugins.utils.routing.semantic_biencoder.semantic_biencoder_routing.SemanticBiEncoderRoutingPlugin Default config: llm_router_plugins/resources/routing/semantic_biencoder.json Trigger: payload[\"model\"] equals auto after trimming ( should_route ) Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_ Dependencies: pip install -e \".[ml]\" (or [ml-gpu] ) What the plugin changes # Key Value model model_name of the winning routing target routing plugin , target_name , similarity (mean cosine of the match) Everything else in the payload is untouched. A rejected match (similarity below threshold) and a request without extractable text leave the payload as it is , so model stays \"auto\" in both cases. { \"model\" : \"qwen3.6:35b\" , \"routing\" : { \"plugin\" : \"semantic_biencoder_routing\" , \"similarity\" : 0.8224 , \"target_name\" : \"code-generation\" } } The log line that explains every decision: SemanticBiEncoderRouting: text='Napisz funkcję w Pythonie, która parsuje plik CSV' target='code-generation' similarity=0.8224 -&gt; model=qwen3.6:35b Quick start # 1. Install with the ML extras # # from a checkout of this repository, into the environment the router runs in pip install -e \".[ml]\" # CPU FAISS + sentence-transformers pip install -e \".[ml-gpu]\" # CUDA FAISS instead — never both FAISS builds at once Without these imports this plugin does not start : the constructor raises and the router refuses to boot. 2. Load the plugin into the router # export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"semantic_biencoder_routing\" Resolved through llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY , instantiated once per process, then run on every prepared payload before guardrails and provider selection. 3. Make the embedding weights available # embedding_model is a Hugging Face id or a local directory . google/embeddinggemma-300m is gated behind the Gemma license, so either accept it and huggingface-cli login , or point at a local copy: { \"embedding_model\" : \"/models/google/embeddinggemma-300m\" } or without touching the file: export LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_MODEL = \"/models/google/embeddinggemma-300m\" A local path also removes the startup download, which matters for air-gapped or restart-heavy deployments. 4. Declare the models in the router model config # the auto alias, backed by a builtin provider (the client asks for it, this plugin replaces it); every model_name used by a routing target ( qwen3.6:35b , gpt-oss:120b , …) as a normal reachable model. // fragment of the llm-router models config \"semantic_routing\": { \"auto\": { \"providers\": [ { \"id\": \"semantic-routing__auto\", \"api_type\": \"builtin\", \"api_host\": null, \"api_token\": \"\", \"model_path\": \"\", \"input_size\": 0, \"weight\": 1.0, \"tool_calling\": false } ] } } 5. Verify # Embedding model loaded successfully. Index built: 6 targets, 79 total embeddings. SemanticBiEncoderRouting: text='…' target='code-generation' similarity=0.8224 -&gt; model=qwen3.6:35b Startup of the shipped config (6 targets, 79 vectors) takes about 9 s on CPU including the model load. See Verifying the plugin for a runnable check. How it works # Startup (constructor) # SemanticBiEncoderConfig.from_file() — bundled JSON, or the …_CONFIG env var (raw JSON string or a path). _override_from_env…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/index.html","t":"Semantic Routing Plugins","s":"Routing plugins","x":"# Semantic Routing Plugins Three routing plugins are available for **model selection** in the **LLM‑Router** system. - `simple_semantic_routing` and `semantic_biencoder_routing` activate when `payload[\"model\"] == \"auto\"`. - `agentic_routin…","h":["Table of Contents","0. Shared Routing Layer","1. Simple Semantic Routing (Heuristic)","1.1 Architecture","1.2 Intent Categories","1.3 Model Selection Algorithm","1.4 Configuration","1.5 Environment Variable Overrides","1.6 Usage Examples","1.7 Weight Tuning Guidelines","1.8 Phrase Format Guidelines","1.9 Complexity Levels","1.10 Built-in Patterns Reference","1.11 Adding New Intents","1.12 Running Tests","2. Bi-Encoder Semantic Routing (Embedding-based)","2.1 Index Building (on first load or when the persist directory is missing)","2.2 Routing (query)","2.3 Persistence","2.4 Configuration","2.5 Usage Examples","2.6 Scoring Details","2.7 Running Tests","3. Codex Routing (Codex CLI Requests)","3.1 Module Layout","3.2 Request Classes","3.3 Detection Cascade","3.4 Similarity: Cosine over Mode Embeddings","3.5 Configuration","3.6 Environment Variable Overrides","3.7 Usage Example","3.8 Running Tests","4. Comparison: Which Plugin to Use?","Recommendation","5. File Locations"],"b":"Semantic Routing Plugins # Three routing plugins are available for model selection in the LLM‑Router system. simple_semantic_routing and semantic_biencoder_routing activate when payload[\"model\"] == \"auto\" . agentic_routing_codex activates when payload[\"model\"] equals its own trigger value ( \"auto_codex\" by default) and routes the Codex CLI's request class (main turn, title generation, compaction) and work mode. Table of Contents # 0. Shared Routing Layer 1. Simple Semantic Routing (Heuristic) 2. Bi-Encoder Semantic Routing (Embedding-based) 3. Codex Routing (Codex CLI Requests) 4. Comparison: Which Plugin to Use? 5. File Locations 0. Shared Routing Layer # semantic_biencoder_routing and agentic_routing_codex share a common layer in llm_router_plugins/utils/routing/ : Module Contents embedder.py EmbeddingRouter (BiEncoder + FAISS: chunking, index build/load/persist, route() ) and EmbeddingRouterConfig — the formal config contract the router duck-types target.py RoutingTarget — the shared target/mode dataclass ( name , model_name , description , examples ); CodexMode is a subclass of it common.py RoutingConfigBase (shared from_file / from_json protocol: ..._CONFIG env var holding raw JSON or a file path, optional default config location), env_int / env_float / env_bool / resolve_persist_dir env helpers, build_embedding_router + check_router_has_vectors , should_route (the payload[\"model\"] trigger gate) and annotate_routing (writes payload[\"model\"] + payload[\"routing\"] ) Backward compatibility: semantic_biencoder/embedder.py and semantic_biencoder/config.py re-export EmbeddingRouter / EmbeddingRouterConfig / RoutingTarget from the shared modules, so existing imports keep working. Both plugins also share the text-extraction helper llm_router_plugins/utils/text_extractor.py::extract_user_text ( messages[-1].content → user_last_statement → query → prompt → input ). 1. Simple Semantic Routing (Heuristic) # The Simple Semantic Routing plugin ( simple_semantic_routing ) performs two-stage heuristic model selection: it classifies the user's intent (code, math, creative, general) via weighted keywords, multi-word phrases, and regex patterns, then estimates input complexity (token count) to pick the most appropriate model from a configured pool. No embedding model is required — routing is a fast, pure-text classification. 1.1 Architecture # Data Flow # User Input │ ▼ classify_intent(text) ← keywords + phrases + regex patterns │ ▼ estimate_tokens(text) ← word_count × 1.25 │ ▼ complexity_level(tokens) ← simple / medium / complex │ ▼ select_model(intent, complexity) ← weighted score → model index │ ▼ Updated payload[\"model\"] Intent Classification Algorithm # Each intent is defined in simple_semantic.json with four complementary signal types : a) Keywords — single words # Each keyword has an optional weight. If no weight is specified, the default is 1.0 . \"keywords\" : [ \"code\" , \"debug\" , \"funkcja\" ], \"weights\" : { \"debug\" : 5 , \"kod\" : 4 , \"code\" : 2 } Matching \"debug\" adds 5 to the score; matching \"code\" adds 2 . b) Phrases — multi-word expressions # Phrases use the format \"text:weight\" . The default weight is 2.0 when omitted. \"phrases\" : [ \"write code:5\" , \"fix bug:4\" , \"napisz funkcję:4\" ] Matching \"write code\" adds 5 to the score. c) Regex Patterns — structural detection # Patterns add a flat +3.0 per match, useful for detecting code structures, math formulas, and question patterns. \"patterns\" : [ \"def\\\\s+\\\\w+\\\\s*\\\\(\" , \"class\\\\s+\\\\w+\\\\s*[(:]\" , \"\\\\b\\\\d+\\\\s*[+\\\\-*/^]\\\\s*\\\\d+\\\\b\" , \"\\\\bwhat\\\\s+(is|are|does)\\\\b\" ] d) Weights — keyword importance # The weights object maps individual keywords to boost factors, allowing fine-grained control: \"weights\" : { \"debug\" : 5 , // Very strong signal for code intent \"błąd\" : 5 , // Very strong signal for code intent \"kod\" : 2 , // Moderate signal \"python\" : 2 // Moderate signal } 1.2 Intent Categories # Intent Description Examples code Programming, debugging, implementation \"napisz funkcję\", \"fix bug…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/routing/simple-semantic/index.html","t":"Simple Semantic Routing (simplesemanticrouting)","s":"Routing plugins","x":"# Simple Semantic Routing (`simple_semantic_routing`) Heuristic, two-stage model selection: it classifies the **intent** of the last user message (code / math / creative / general) with weighted keywords, phrases and regexes, estimates the…","h":["What the plugin changes","Quick start","1. Install","2. Load the plugin into the router","3. Declare the models in the router model config","4. Verify","How it works","Where it runs","Text extraction","Stage 1 — intent classification","Stage 2 — complexity estimation","Stage 3 — model selection","Configuration","Config file layout","Environment variables","Validation","Tuning and gotchas","Verifying the plugin","Offline dry run","Router smoke test","Automated tests","Troubleshooting","Reference","Defaults at a glance","Files","See also"],"b":"Simple Semantic Routing ( simple_semantic_routing ) # Heuristic, two-stage model selection: it classifies the intent of the last user message (code / math / creative / general) with weighted keywords, phrases and regexes, estimates the complexity from a rough token count, and picks a model from an ordered pool. No embedding model, no vector store, no network — the whole decision is a few string scans, so it costs microseconds and is fully reproducible. It is the entry-level routing plugin: turn model: \"auto\" into \"a sensible model for this request\" without any ML dependencies. When you outgrow keyword heuristics, move to semantic_biencoder_routing , which decides by embedding similarity instead. Plugin name (registry key): simple_semantic_routing Class: llm_router_plugins.utils.routing.simple_semantic.simple_semantic_routing.SimpleSemanticRoutingPlugin Default config: llm_router_plugins/resources/routing/simple_semantic.json Trigger: payload[\"model\"] == \"auto\" (exact match) Env prefix: LLM_ROUTER_ROUTING_SEMANTIC_ Dependencies: none beyond the base package What the plugin changes # apply() sets exactly one key: Key Value model the pool entry selected for this intent + complexity Nothing else is added — there is no routing metadata block in the payload (unlike semantic_biencoder_routing and agentic_routing_codex ). Everything the plugin knows is written to the log: SimpleSemanticRouting: intent=code, complexity=simple (10 tokens) -&gt; qwen3.6:35b If no user text can be found, model is set to the configured default model and a warning is logged; a payload whose model is not exactly \"auto\" is returned untouched. Quick start # 1. Install # # from a checkout of this repository, into the environment the router runs in pip install -e . No extras needed — this plugin imports only the standard library. 2. Load the plugin into the router # export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"simple_semantic_routing\" The router resolves the name in llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY , instantiates it once per process and runs apply() on every prepared payload, before guardrails and before a provider is selected. 3. Declare the models in the router model config # Everything the plugin can select must exist in the llm-router model config ( LLM_ROUTER_MODELS_CONFIG ): the auto alias itself, backed by a builtin provider — the client asks for it, the plugin replaces it; every model in the pool ( default_models values, or the LLM_ROUTER_ROUTING_SEMANTIC_MODELS list). // fragment of the llm-router models config \"semantic_routing\": { \"auto\": { \"providers\": [ { \"id\": \"semantic-routing__auto\", \"api_type\": \"builtin\", \"api_host\": null, \"api_token\": \"\", \"model_path\": \"\", \"input_size\": 0, \"weight\": 1.0, \"tool_calling\": false } ] } } A pool entry the router does not know is a hard downstream error, not a routing no-op. 4. Verify # Send a request with \"model\": \"auto\" and look for the log line, or run the verification snippet . Two lines to remember: SimpleSemanticRouting: intent=code, complexity=simple (10 tokens) -&gt; qwen3.6:35b No text content found, using default model gpt-oss:120b How it works # payload ──▶ extract text ──▶ stage 1: intent ──▶ stage 2: complexity ──▶ stage 3: pool index ──▶ payload[\"model\"] Where it runs # Inside the llm-router utils pipeline. The plugin is stateless: no cache, no session, no learned state, so every replica answers identically. Text extraction # Text is located by the shared helper llm_router_plugins.utils.text_extractor.extract_user_text , in this priority: payload[\"messages\"][-1][\"content\"] — the last message of a chat history (earlier turns are not seen) payload[\"user_last_statement\"] payload[\"query\"] payload[\"prompt\"] payload[\"input\"] Only that one string is scored, so intent detection always reflects the newest user turn. Stage 1 — intent classification # Every intent in intents accumulates a score from three signal kinds; the highest score above zero wins, ties go to the intent declared first in …","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/knowledge/agentic-knowledge/index.html","t":"Agentic knowledge layer (agentic_knowledge)","s":"Utility plugins","x":"# Agentic knowledge layer (`agentic_knowledge`) A knowledge layer for coding agents that lives **in the proxy, with no cooperation from the agent**. It reads the work state the routing plugin has already derived (work mode, phase evidence,…","h":["What the plugin changes","Quick start","1. Install","2. Load it into the pipeline, after routing","3. Point it at SSE and Redis","4. Run the indexing worker","5. Verify","How it works","Scope","Collections and documents","Read path: retrieval per work mode","Generation cache","Write path","Compaction archive","Documentation (doc)","Standing documentation context (doc_context)","Linking knowledge to documentation","Knowledge-or-escalate","Annotation","Configuration","How one request reads the engine","Environment variables","Worker environment","Documentation command","Adapters","Codex (full)","Claude Code (basic)","Privacy, masking, retention","Stateless mode","Evaluation","Limitations","See also"],"b":"Agentic knowledge layer ( agentic_knowledge ) # A knowledge layer for coding agents that lives in the proxy, with no cooperation from the agent . It reads the work state the routing plugin has already derived (work mode, phase evidence, request class), retrieves what this repository has already learned from the Semantic Search Engine (SSE), injects it into the request, and learns from the traffic itself: verified fixes, dead ends, and the window that a compaction is about to throw away. The retrieval result becomes a routing signal as well — an agent spinning in a loop with no known fix is moved to a stronger model ( knowledge-or-escalate ). Plugin name (registry key): agentic_knowledge Class: llm_router_plugins.utils.knowledge.agentic_knowledge.plugin.AgenticKnowledgePlugin Default config: llm_router_plugins/resources/knowledge/agentic_knowledge.json Env prefix: LLM_ROUTER_KNOWLEDGE_AGENTIC_ Worker: llm-router-agentic-kb-worker Requires: the SSE client and Redis — both optional, see Install It is designed to stand behind agentic_routing_codex in the utils pipeline: it reads payload[\"routing\"] , payload[\"agent_mode\"] and payload[\"model\"] as the router left them, and never touches the classifier. What the plugin changes # Key Value input / system one extra context block when knowledge was found (adapter-specific) model the escalation model, only when the escalation policy fired routing.knowledge annotation of what this plugin did (see Annotation ) routing.model_before_knowledge previous model , only when this plugin replaced it Everything else is forwarded unchanged. The plugin never rejects a request. Fail-open everywhere except the config file. A non-dict payload, an unclaimed payload, a parse error, an SSE timeout, SSEAPIError , a missing sse_lib , a Redis outage — all return the payload as it came, with at most one warning that carries no document content. Malformed Redis connection settings disable the features that need Redis and keep the policy. A malformed policy (unknown key, out-of-range weight, non-positive budget) raises at startup, the same fail-hard rule the Codex router uses. Quick start # 1. Install # pip install -e \".[agentic-knowledge]\" # sse-lib&gt;=0.1.0 + redis&gt;=5 sse_lib depends only on requests . Without it, and without a configured host, the plugin loads and does nothing. Documentation uploads require both the client and SSE server to support the upload_files(file_metadata=...) contract; update both together. Older clients cannot preserve documentation metadata through file uploads. 2. Load it into the pipeline, after routing # export LLM_ROUTER_UTILS_PLUGINS_PIPELINE = \"agentic_routing_codex,agentic_knowledge\" Knowledge needs the routing decision, so it must run after the routing plugin. Registered in MAIN_UTILS_REGISTRY under agentic_knowledge ; instantiated once per process. 3. Point it at SSE and Redis # export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_HOST = \"https://sse.internal\" # A pre-issued token, or a login for an engine that authenticates by username and password # (the client logs in on its first call; a token, when set, wins). export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_TOKEN = \"…\" export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_USERNAME = \"…\" export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_PASSWORD = \"…\" export LLM_ROUTER_KNOWLEDGE_AGENTIC_REDIS_HOST = \"redis.internal\" No SSE_API_HOST ⇒ no reads (requests pass through). No REDIS_HOST ⇒ no writes , and the generation cache falls back to one that lives in the process only (reads stay stateless across workers, see Stateless mode ). 4. Run the indexing worker # export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_EMBEDDER = \"&lt;embedder served by your SSE&gt;\" export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_RERANKER = \"&lt;reranker served by your SSE&gt;\" llm-router-agentic-kb-worker Without a worker the layer only reads: nothing new is ever learned. The embedder and reranker are required because the worker creates a collection the first time a scope produces knowledge. T…","r":"llm-router-plugins"},{"k":"llm-router-plugins/utils/rag/langchain-rag.html","t":"LangChain RAG Plugin for LLM‑Router","s":"Utility plugins","x":"# LangChain RAG Plugin for LLM‑Router A lightweight **Retrieval‑Augmented Generation (RAG)** plugin that lets you attach any local knowledge base to the LLM‑Router without touching the original application code. It builds the knowledge bas…","h":["Table of Contents","Features","Installation","CLI – Indexing &amp; Searching","Searching","Example workflow","Error handling","License &amp; Credits","Happy augmenting! 🚀"],"b":"LangChain RAG Plugin for LLM‑Router # A lightweight Retrieval‑Augmented Generation (RAG) plugin that lets you attach any local knowledge base to the LLM‑Router without touching the original application code. It builds the knowledge base with LangChain , stores vectors in FAISS (fallback to Milvus is possible), and injects the retrieved context into every LLM request. Table of Contents # Features Installation Configuration (environment variables) CLI – Indexing &amp; Searching Integrating the plugin into LLM‑Router Example workflow Error handling License &amp; Credits Features # Feature Description Indexing Recursively reads a directory of .md , .txt , .html (or any extensions you pass) → splits documents into overlapping token windows → embeds chunks with a transformer model → stores them in a persistent FAISS index. Searching Given a user query, retrieves the top‑N most similar chunks (default 10) using cosine similarity. Context injection Retrieved chunks are appended (with a clear prefix) to the user’s last message, so the downstream LLM can use the extra knowledge automatically. Full configurability All hyper‑parameters (collection name, embedder model, device, chunk size, overlap, persistence directory) are set via environment variables. Fail‑safe If the RAG subsystem is disabled or mis‑configured, the plugin raises a clear exception while the rest of the router keeps working. CLI helpers llm-router-rag-langchain command provides convenient index and search sub‑commands, plus an interactive REPL mode. Installation # Only the optional RAG dependencies are required; they are not installed with the core LLM‑Router. ```shell script Activate your virtualenv / conda env first # pip install faiss-cpu tqdm langchain langchain-community transformers torch &gt; **Note** – If you prefer GPU‑accelerated FAISS, replace `faiss-cpu` with `faiss-gpu`. --- ## Configuration (environment variables) | Variable | Default | Description | |------------------------------------------|---------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | `LLM_ROUTER_LANGCHAIN_RAG_COLLECTION` | *must be set* | Name of the FAISS collection (e.g. `sample_collection`). | | `LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER` | *must be set* | Hugging Face model identifier for the embedder (e.g. `sentence-transformers/all-MiniLM-L6-v2`). | | `LLM_ROUTER_LANGCHAIN_RAG_DEVICE` | `cpu` | Torch device (`cpu`, `cuda:0`, `cuda:1`, …). | | `LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE` | `1024` | Number of tokens per chunk. | | `LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP` | `100` | Overlap in tokens between consecutive chunks. | | `LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR` | *none* | Directory where the FAISS index and docstore are persisted. If omitted, a folder named after the collection is created under `./workdir/plugins/utils/rag/langchain/`. | ### Example export block (add to your shell profile or a `.env` file) ```shell script export LLM_ROUTER_LANGCHAIN_RAG_COLLECTION=\"${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION:-sample_collection}\" export LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER=\"${LLM_ROUTER_LANGCHAIN_RAG_EMBEDDER:-sentence-transformers/all-MiniLM-L6-v2}\" export LLM_ROUTER_LANGCHAIN_RAG_DEVICE=\"${LLM_ROUTER_LANGCHAIN_RAG_DEVICE:-cpu}\" export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE=\"${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_SIZE:-200}\" export LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP=\"${LLM_ROUTER_LANGCHAIN_RAG_CHUNK_OVERLAP:-50}\" export LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR=\"${LLM_ROUTER_LANGCHAIN_RAG_PERSIST_DIR:-./workdir/plugins/utils/rag/langchain/${LLM_ROUTER_LANGCHAIN_RAG_COLLECTION}}\" CLI – Indexing &amp; Searching # The plugin ships with a convenient entry‑point: ```shell script llm-router-rag-langchain [options] ### Indexing ```shell script # Index Markdown, plain‑text and HTML files from a repository llm-router-rag-langchain index --path \"../llm-router\" --ext .md .txt .html You can run …","r":"llm-router-plugins"},{"k":"scripts/plugins/auto-codex/eval/index.html","t":"Codex routing — offline corpus and replay","s":"Plugin scripts","x":"# Codex routing — offline corpus and replay Tools for evaluating `agentic_routing_codex` against **recorded Codex CLI traffic** instead of synthetic payloads. Nothing here calls a model provider: the corpus is replayed through the routing …","h":["Pipeline","What replay.py guarantees","Reading compare_history.py","Known limits"],"b":"Codex routing — offline corpus and replay # Tools for evaluating agentic_routing_codex against recorded Codex CLI traffic instead of synthetic payloads. Nothing here calls a model provider: the corpus is replayed through the routing code in the checkout, so a change in the cascade shows up as a change in the numbers rather than being blended with the decisions the recording server happened to make. Recorded traffic lives in agents-conversation/codex/ (router debug logs containing full request bodies). The corpus and decision files are generated artifacts — keep them in /tmp or another scratch directory, they are large (793 requests ≈ 180 MB of JSON). Pipeline # EVAL = scripts/plugins/auto_codex/eval # 1. Recover payloads and the decision logged for each of them python3 $EVAL /extract_payloads.py -o /tmp/corpus.jsonl # 2. Replay the current code over the corpus python3 $EVAL /replay.py /tmp/corpus.jsonl -o /tmp/decisions.jsonl --determinism # 3. Compare with the recorded decisions (diagnostic, not a score) python3 $EVAL /compare_history.py /tmp/decisions.jsonl --recorded --show 5 # 4. Look at a single disagreement python3 $EVAL /inspect_case.py /tmp/corpus.jsonl /tmp/decisions.jsonl \\ --line 104387 replay.py --semantic adds the embedding layer (loads the configured sentence-transformers model, ~25 s of startup). Run both variants: the deterministic one is what a semantic.enabled=false deployment ships. Against a labeled corpus, use the in-tree evaluation instead — it measures accuracy, while this directory measures what moved: python3 -m llm_router_plugins.utils.routing.agentic_routing.codex.evaluation \\ --dataset tests/data/codex_routing_quality.json --split all --no-semantic What replay.py guarantees # Three invariants are reported on every run, because a routing change must never break them: apply() raises nothing — routing fails open; a payload that was not routed comes back as the same object , byte for byte; a routed payload gains exactly agent_mode and routing ( annotate_routing mutates in place, so the pre-call key set is captured before the call). --determinism replays the corpus backwards through a second plugin instance and checks the decisions are identical, which catches any hidden dependence on request order. Reading compare_history.py # Roughly half the requests disagree with the recording, and that is expected: the logs span several versions of the cascade and of the configuration (an earlier deployment ran with memory and the semantic layer enabled, the shipped config is stateless). Treat it as a change-impact and case-mining tool: agreement per source file localises where behaviour moved; the recorded -&gt; current confusion table names the pairs worth a human look; --list-unrouted separates a coverage change (the request is no longer routed at all) from a wrong mode, so a non-trigger payload paired with a neighbouring decision cannot fake the agreement number. Correctness claims belong to the labeled corpus and to the metrics in codex/evaluation.py — per-mode precision/recall and transition counts. Known limits # Requests are paired with decisions in stream order . A payload whose decision line never arrived keeps recorded: null , and a payload interleaved with another request's decision can be matched to the wrong one; the two visible cases are reported by --list-unrouted . extract_payloads.py needs router logs at DEBUG / INFO with the request body printed as indented JSON — a one-line body is not recovered. Log blocks interrupted by another log line are counted and skipped ( unparsable blocks dropped ), never guessed.","r":"llm-router-plugins"},{"k":"scripts/plugins/auto-codex/payload-parser/index.html","t":"Codex routing — per-request payload export","s":"Plugin scripts","x":"# Codex routing — per-request payload export Turns the recorded router traffic in `agents-conversation/` into **one JSON document per request**: the request exactly as the Codex CLI sent it, plus the mode and model the plugin in this check…","h":["Usage","Document shape","Resuming"],"b":"Codex routing — per-request payload export # Turns the recorded router traffic in agents-conversation/ into one JSON document per request : the request exactly as the Codex CLI sent it, plus the mode and model the plugin in this checkout decides for it. Parsing is shared with ../eval — one recovery rule for both entry points — and the decision always comes from the current code, never from the line the recording server logged. Usage # # whole recorded corpus, deterministic cascade python3 scripts/plugins/auto_codex/payload_parser/export_payloads.py # one session, embedding layer on, into a scratch directory python3 scripts/plugins/auto_codex/payload_parser/export_payloads.py \\ agents-conversation/codex/conv-03 --semantic -o /tmp/payloads python3 scripts/plugins/auto_codex/payload_parser/export_payloads.py --help Output is agents-conversation/codex/payloads/codex_payload_NNN.json , numbered in recovery order (log order). The directory is gitignored — it is hundreds of MB of derived data. Document shape # { \"id\": 63, \"source\": \"agents-conversation/codex/conv-03/llm-router.log\", \"line\": 1842, \"logged_at\": \"2026-10-09 04:57:14,037\", \"trigger\": \"auto_codex\", \"routed\": true, \"decision\": { // what the current code decided \"mode\": \"git_review\", // work mode \"model\": \"qwen/Qwen3.8-27B\", // chosen model \"layer\": \"phase\", // which layer answered \"similarity\": 1.0, \"class\": \"main\", // main | aux_title | compaction \"reason\": \"git log\", \"evidence\": \"command\", \"memory\": null, \"semantic\": null, \"error\": null // set only if routing raised (it must not) }, \"routing\": { ...raw plugin annotation... }, // same decision, unflattened \"recorded\": { \"mode\": \"implement\", \"source\": \"fallback\", ... } | null, \"payload\": { ...request as sent, untouched... } } payload is the request before routing: the plugin rewrites model and adds routing / agent_mode in place, so a copy is routed and the original is what gets stored. Compare decision.model with payload.model to see the rewrite. recorded is the decision the recording server made, kept for diffing; null when the log had none. Requests the plugin does not own (model other than the trigger) are written with routed: false , decision.model equal to the model they arrived with and decision.mode null; --only-trigger skips them. Resuming # A run can be interrupted and repeated: file NNN is reused unless --overwrite is given, and the stored source / line are checked against the request that would be written there. Changing the filters changes the numbering, so a mismatch aborts with a message instead of silently relabelling already written documents — pass --overwrite or an empty -o directory then.","r":"llm-router-plugins"},{"k":"changelog.html","t":"Changelog","s":"Other plugins docs","x":"# Changelog All notable changes to the `llm-router-plugins` package are documented here. Versions follow the `MAJOR.MINOR.PATCH` scheme; a new entry is added for every release tagged as `v<version>`. ## Summary | Version | Changelog | | --…","h":["Summary","[Unreleased]","Added","Changed","Fixed","[0.1.4] - 2026-10-21","Added","Changed","Fixed","[0.1.3] - 2026-10-11","Added","Changed","Fixed","[0.1.2] - 2026-10-08","Added","Changed","[0.1.1] - 2026-09-25","Added","Changed","[0.1.0] - 2026-09-06","Changed","[0.0.6] - 2026-08-31","Added","Changed","[0.0.5] - 2026-07-16","Added","Changed","[0.0.2] - 2026-05-10","Added","Changed","[0.0.1] - 2025-11-28","Added"],"b":"Changelog # All notable changes to the llm-router-plugins package are documented here. Versions follow the MAJOR.MINOR.PATCH scheme; a new entry is added for every release tagged as v&lt;version&gt; . Summary # Version Changelog 0.1.4 Added: agentic_knowledge — a knowledge layer that runs behind the Codex router: per-work-mode retrieval of verified fixes, dead ends and the pre-compaction archive from the Semantic Search Engine, a masked learn-from-traffic write path through a Redis Stream and an indexing worker, and optional knowledge-or-escalate model selection. Adapters for Codex (full) and Claude Code (read-only), plus offline counterfactual evaluation. Fix: the agentic_knowledge collection names are reduced to the engine's character set at configuration time, so a repository scope that would produce a hyphen can no longer make every indexing attempt fail with a 500 at collection creation; the retention pass still sweeps the legacy prefix-&lt;scope&gt;-sessions archives; the repository of a request is read from the leading messages, so a compaction (which re-sends the whole window) is filed under its own project instead of into a scope-less prefix, and a request with no known repository publishes nothing; a search of a repository that has not been indexed yet is an empty answer, not a failed search Added: engine login for a Semantic Search Engine that authenticates by username and password rather than by a pre-issued token ( SSE_API_USERNAME / SSE_API_PASSWORD ); every mined episode now carries the work phase its evidence names ( test , implement , …) in its header and in metadata_json.phase , and a retrieval source can be restricted in its policy to phases ( \"phases\": [\"test\"] ) or to routing modes ( \"modes\": [\"debug\"] ) — the filter is engine-side, unions across names, intersects the source's categories, never touches the archived window whose thread_id owns the single metadata slot, and a policy cannot mix the two dimensions in one request. Added: agentic_knowledge indexes the repository's own documentation — a heading-aware scan of its *.md files published as doc documents, republished on content change, superseded documents retired through a docs_prune control message (never on a truncated scan), plus the llm-router-agentic-kb-docs command and the docs policy section. Documentation now lives in &lt;prefix&gt;_&lt;scope&gt;_docs , is read per work mode through doc_types , is searched on the turn's subject rather than its failure signature, and a guide about a file the turn edits outranks one that is not. On top of retrieval a session carries a standing documentation map — structural, stable per repository, budgeted first — and every mined fix or dead end points at the page that documents the module it touched. Fix: the Codex routing record follows the phase that was actually routed with instead of the last signal it could rebuild, so an interleaved command no longer freezes the record and repeats a conflict on every request of the turn (captured-session replay: 11 conflicts -&gt; 0). Added: RememberResult carries why a memory write was skipped; the log names the real cause instead of an undifferentiated version conflict . Added: agentic_knowledge mines playbook — a change a test run accepted with no failure in between — so the correct steps of a task are knowledge too, not only the ones that went wrong; a mode routing did not resolve is served by a * fallback policy instead of reading nothing; every mode now reads the kinds it ignored ( doc in debug / test , deadend in implement , playbook / recipe where a plan or a review could use them). Fix: a read budget equal to one engine call started no search of any request — the deadline left no room for the call it budgeted — which looked like an empty knowledge base; the budget is now raised above one call at validation, defaults to three calls, and the \"searches not started\" warning names both numbers and repeats at most once per repository per interval. 0.1.3 Added: an acc…","r":"llm-router-plugins"}]}