Codex CLI Routing (agentic_routing_codex)#
Routing plugin for the OpenAI-Responses-style requests emitted by the Codex CLI coding agent. It looks at
every request that arrives with the trigger model (auto_codex by default), decides what kind of work the agent is
doing right now, and rewrites payload["model"] to the model configured for that work mode. Planning turns, code
reviews, test runs, debugging, one-line thread titles and context compaction can then each run on the model that fits
them best, while the Codex CLI keeps talking to a single stable model name.
The decision cascade is deterministic first: structural signals (request class, the <collaboration_mode> block
the CLI injects, keyword scoring) answer most requests offline, for free, and reproducibly. An optional embedding layer
contributes cosine similarity over the mode descriptions and examples only when the cheap layers stay silent.
- Plugin name (registry key):
agentic_routing_codex - Class:
llm_router_plugins.utils.routing.agentic_routing.codex.plugin.CodexRoutingPlugin - Default config:
llm_router_plugins/resources/routing/agentic_routing_codex.json - Env prefix:
LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_
What the plugin changes#
apply() rewrites the payload in place and touches exactly three keys:
| Key | Value |
|---|---|
model |
model_name of the resolved Codex work mode |
agent_mode |
name of the resolved mode (plan, implement, test, …) |
routing |
decision metadata: plugin, similarity, source, class and ids |
Everything else — input, instructions, tools, reasoning, text, stream, parallel_tool_calls — is forwarded
unchanged. The plugin never rejects a request; it only ever picks a model.
Fail-open by design. A payload that is not a dict, does not carry the trigger model, resolves to a mode that is not
configured or to a mode with an empty model_name, or that raises anywhere during parsing or classification, is
returned untouched with the trigger model still in place. Fail-hard on configuration only: an inconsistent config
raises at plugin construction, so the router refuses to start instead of silently misrouting every request.
Quick start#
1. Install the plugin package#
# from a checkout of this repository, into the environment the router runs in
pip install -e . # deterministic routing only
pip install -e ".[ml]" # + CPU FAISS and sentence-transformers (semantic layer)
pip install -e ".[ml-gpu]" # + CUDA FAISS instead — never both FAISS builds at once
The [ml] / [ml-gpu] extras install faiss-cpu/faiss-gpu, sentence-transformers, numpy and scipy. Without
them the plugin still loads and routes deterministically — the semantic layer steps aside with a single warning. In a
container image, add the package (with the extra you want) next to llm-router so both import the same
sentence-transformers.
2. Load the plugin into the router#
The plugin is a utils plugin: it runs inside the llm-router request pipeline and is selected by a comma-separated list of plugin identifiers.
export LLM_ROUTER_UTILS_PLUGINS_PIPELINE="agentic_routing_codex"
The router resolves the name against llm_router_plugins.utils.registry.MAIN_UTILS_REGISTRY, instantiates it once per
process (UtilsRegistry), and calls its apply() on every prepared request, before guardrails and before a provider
is chosen. Order matters if you wire several utils plugins together: the first plugin that rewrites model wins the
downstream dispatch.
Instantiating the plugin loads the JSON config, applies env overrides, validates them and — when the semantic layer is enabled — loads the embedding model and builds (or loads) the FAISS index. Expect a few seconds of extra startup time and a couple of hundred MB of RSS for a 300 M embedding model.
3. Declare the models in the router#
Two things must exist in the llm-router model config (LLM_ROUTER_MODELS_CONFIG, JSON):
- The trigger
auto_codexas a declared model alias backed by abuiltinprovider — the CLI asks for it, the router accepts it, and the plugin replaces it before any upstream call is made. It follows the same pattern as theautoalias used by the semantic routing plugins:
jsonc
// fragment of the llm-router models config (LLM_ROUTER_MODELS_CONFIG)
"semantic_routing": {
"auto_codex": {
"providers": [
{
"id": "semantic-routing__auto_codex",
"api_type": "builtin",
"api_host": null,
"api_token": "",
"model_path": "",
"input_size": 0,
"weight": 1.0,
"tool_calling": true
}
]
}
}
- Every model referenced by
codex_modes[].model_nameas a normal, active model with a reachable provider (api_type: "vllm",ollama,openai, …). A name the router does not know is a hard downstream failure (model not found), not a routing error: the plugin has already done its job.
4. Point the Codex CLI at the router#
~/.codex/config.toml:
model = "auto_codex"
model_provider = "llm_router"
model_context_window = 256000
model_max_output_tokens = 32000
[model_providers.llm_router]
name = "LLM Router"
base_url = "http://<router-host>:8081/v1"
wire_api = "responses"
env_key = "LLM_ROUTER_API_KEY"
wire_api = "responses" is what produces the /v1/responses request shape this plugin reads, and LLM_ROUTER_API_KEY
must hold a valid router API key. Because auto_codex is a stable alias, nothing else changes when you retune the
mode→model mapping — no CLI redeploy, no git pull of client configs.
5. Verify#
Two log lines prove the plugin is loaded and deciding:
[utils] Registered utility plugin 'agentic_routing_codex' as instance 'CodexRoutingPlugin'
Codex routing: mode=plan source=collaboration_mode model=qwen/Qwen3.8-Flash-Next similarity=1.000 class=main
For an end-to-end check without the CLI, see Verifying the plugin.
How it works#
Where it runs#
Codex CLI ──POST /v1/responses (model=auto_codex)──▶ llm-router
auth middleware → prepare_payload → UtilsPipeline → guardrails → masking → provider dispatch ──▶ vLLM
│
└── CodexRoutingPlugin.apply(payload)
The plugin sees the prepared request payload as a plain dict and returns it. It is stateless: no session cache, no
memory between requests, so the same request is routed identically by any replica at any time.
Step 1 — payload normalization#
parse_codex_payload() is the only code that knows where Codex metadata lives on the wire; everything downstream works
on an immutable CodexRequest snapshot. Reading it never raises — an unreadable value keeps the field default.
| Snapshot field | Read from |
|---|---|
session_id, thread_id, turn_id, root_turn_id |
client_metadata.* (turn metadata as fallback) |
window_id |
client_metadata["x-codex-window-id"] |
request_kind, thread_source, sandbox_mode, agent_name, context_window_id |
client_metadata["x-codex-turn-metadata"], which is a JSON-encoded string |
tool_names, has_tools |
tools[].name (falls back to type for built-in tools) |
structured_output |
text.format.type == "json_schema" |
reasoning_effort, parallel_tool_calls |
reasoning.effort, parallel_tool_calls |
context_chars, context_tokens |
content length of instructions + input (tokens ≈ chars / 4) |
collaboration_mode |
<collaboration_mode>…</collaboration_mode> block in developer messages |
latest_user_text |
genuine user messages, newest first, within classify_max_chars |
Details that matter when you debug routing:
- Collaboration mode is read from every
role == "developer"message; the last<collaboration_mode>block wins, so a Plan → Default switch mid-session is picked up on the next turn. Its first heading decides:# Plan Mode→plan,# Collaboration Mode: Default→default, anything else →"". latest_user_textscansinputin reverse and keeps onlytype == "message",role == "user"items whose text is not just an<environment_context>block. The newest message is always kept whole; older ones are appended while theclassify_max_charsbudget holds (a non-positive budget means "unbounded"). If the request carries no usable user message, the legacypayload["prompt"]is used.
Step 2 — request class#
Codex mixes three kinds of request on one endpoint. The structural ones are recognised before any prompt text is
read, with strict priority compaction > aux_title > main:
| Class | Condition | Why it matters |
|---|---|---|
compaction |
request_kind == "compaction" |
Carries the whole transcript; wins over every keyword signal |
aux_title |
request_kind == "turn" and thread_source == "system" |
CLI-generated one-line thread title, no tools, JSON-schema output |
main |
everything else | A regular agent turn (Plan or Default collaboration mode) |
Step 3 — resolution cascade#
classify() tries layers in a fixed order; the first layer that can answer wins. A main turn never falls below the
fallback mode: the keyword and semantic layers can specialise the decision, never weaken it.
| # | Layer | routing.source |
Confidence |
|---|---|---|---|
| 1 | Explicit agent_mode / codex_mode / metadata.agent_mode in the payload |
explicit |
1.0 |
| 2 | Request class — compaction, then aux_title |
class |
1.0 |
| 3 | <collaboration_mode> Plan block from the CLI |
collaboration_mode |
1.0 |
| 4 | Keyword scoring of the user text | heuristic |
score / (score + 1) |
| 5 | Embedding cosine similarity over the modes | semantic |
cosine of the matched mode |
| 6 | Configured fallback_mode (implement) |
fallback |
cosine of that mode, else 0.0 |
Notes:
- An explicit override must name a configured mode, otherwise it is ignored and the cascade continues;
agent_modeis checked beforecodex_mode, and both beforemetadata.agent_mode. This is the escape hatch for "route this one request by hand": send"agent_mode": "debug"in the request body. - Only four modes compete in the keyword layer, scanned in the order
test,git_review,review,debug(HEURISTIC_MODES).planis declared by the CLI andimplementis the fallback, so theirkeywords/phrases/patternsare never scored — only theirdescriptionandexamplesfeed the embedding index.aux_titleandcompactionare class-routed and are left out of the index too. - Layers 1–4 never touch the embedding stack. The vector store is queried at most once per request, and only after the deterministic layers have stayed silent.
Step 4 — keyword scoring#
Matching is case-insensitive (DEBUG matches debug) and keywords and phrases must start at a word boundary. A
declared stem therefore also hits inflected forms — test matches testy and testów, napraw matches
naprawiłem — while mid-word occurrences are ignored (protest and kontest do not match test). Letters are
not diacritic-folded, which is exactly why the shipped lists carry błąd and blad, nie działa and
nie dziala, plan rozwiązania and plan rozwiazania.
| Signal | Weight | Example |
|---|---|---|
keywords |
weights[keyword], else 1.0 |
"debug": 3 |
phrases |
":weight" suffix, else 2.0 |
"napraw błąd:3" |
patterns |
3.0 per match |
"\broot\s+cause\b" |
Scores of all matched signals are summed per mode; detect_mode uses strictly-greater-than while scanning
HEURISTIC_MODES in order, so ties go to the earlier mode (test over git_review, git_review over review).
The best score is accepted only when it reaches heuristic_min_score (3.0 by default), and is reported as
similarity = score / (score + 1).
Measured against the shipped configuration:
| Text | Mode | Score | Accepted (≥ 3.0) |
|---|---|---|---|
napraw testy w tests/ |
test |
14.0 | yes → 0.933 |
testy |
test |
8.0 | yes → 0.889 |
Dlaczego nie działa ta funkcja? |
debug |
9.0 | yes → 0.900 |
review this PR |
review |
9.0 | yes → 0.900 |
git review przed merge |
git_review |
11.0 | yes → 0.917 |
Przygotuj plan refactoru |
review |
2.0 | no → semantic/fallback |
Dodaj nowy endpoint do API |
— | 0.0 | no → semantic/fallback |
The last two rows are the point of the design: plan is never decided by keywords (the CLI tells you, in Plan
Mode, and a planning sentence in Default mode is not enough), and a plain imperative like "add an endpoint" is left to
the semantic layer or to fallback_mode — which is exactly what implement means.
Step 5 — semantic similarity (optional)#
Enabled with settings.semantic.enabled, disabled for a single process with
LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_SEMANTIC_ENABLED=false.
- An
EmbeddingRouterindexesdescription+examplesof the six non-class-routed modes with a sliding window (chunk_size256,chunk_overlap64) using a sentence-transformers BiEncoder. - Index vectors are L2-normalized and searched with FAISS inner product, which is cosine similarity.
- A query longer than the embedding model's
max_seq_lengthis split into overlapping token windows; at most the firstMAX_QUERY_WINDOWS = 4windows are encoded and their unit vectors are averaged and renormalized, so a huge prompt costs a bounded amount of compute. top_khits are grouped per mode and averaged; the best mode wins and is accepted only when its average cosine reachessimilarity_threshold(0.51by default).- One lookup serves both the accept decision and the fallback mode's reported similarity, so a request never embeds the same text twice.
- The model is loaded with
device="cpu"andtrust_remote_code=Trueat plugin construction. A router that raises at query time (broken index, empty vector) disables only the semantic answer for that request.
Step 6 — annotating the payload#
Before (trimmed real Codex turn, Plan Mode):
{
"model": "auto_codex",
"instructions": "You are a coding agent running in the Codex CLI.",
"input": [
{"type": "message", "role": "developer", "content": [{"type": "input_text",
"text": "You are a coding agent running in the Codex CLI.\n\n<collaboration_mode># Plan Mode (Conversational)\n\nThink, do not edit.</collaboration_mode>"}]},
{"type": "message", "role": "user", "content": [{"type": "input_text",
"text": "<environment_context>\n <cwd>/repo</cwd>\n</environment_context>"}]},
{"type": "message", "role": "user", "content": [{"type": "input_text",
"text": "Zaplanuj migrację bazy danych."}]}
],
"tools": [{"type": "function", "name": "exec_command", "parameters": {}}],
"parallel_tool_calls": true,
"reasoning": {"effort": "medium", "summary": "auto"},
"stream": true,
"client_metadata": {
"thread_id": "thread-1",
"turn_id": "turn-7",
"x-codex-window-id": "thread-1:0",
"x-codex-turn-metadata": "{\"request_kind\":\"turn\",\"thread_source\":\"user\",\"sandbox_mode\":\"danger-full-access\",\"agent_name\":\"/root\"}"
}
}
After (model replaced, agent_mode and routing added):
{
"model": "qwen/Qwen3.8-Flash-Next",
"agent_mode": "plan",
"routing": {
"plugin": "agentic_routing_codex",
"similarity": 1.0,
"agent_mode": "plan",
"source": "collaboration_mode",
"codex_class": "main",
"collaboration_mode": "plan",
"request_kind": "turn",
"thread_id": "thread-1",
"turn_id": "turn-7"
}
}
routing is metadata for logs, tracing and cost attribution. If your upstream is strict about unknown fields, strip
it after logging — the plugin itself only guarantees that the request shape Codex expects is preserved.
Failure behaviour#
| Situation | Result |
|---|---|
| payload is not a dict | same object, untouched |
model absent, not a string, or ≠ trigger (after strip()) |
same object, untouched |
| parse or classify raises | same object, untouched + warning log |
| resolved mode is not configured | same object, untouched + warning log |
resolved mode has model_name: "" |
same object, untouched + warning log |
| FAISS / sentence-transformers missing while semantic is enabled | plugin loads, semantic layer off, warning log |
| semantic lookup raises at query time | that request continues without semantic |
empty trigger, no modes, duplicate mode names, unknown fallback_mode, semantic on with no embedding model, chunk_size <= 0, chunk_overlap < 0, top_k < 1, classify_max_chars <= 0 |
ValueError at construction — router refuses to start |
Configuration#
Config file layout#
{
"description": "what this config is for (documentation only)",
"embedding_model": "/models/google/embeddinggemma-300m", // top level, NOT in settings
"embedding_model_public": "google/embeddinggemma-300m", // informational, never read
"settings": {
"trigger_model": "auto_codex", // required
"fallback_mode": "implement", // required, must name a mode below
"heuristic_enabled": true,
"heuristic_min_score": 3.0,
"classify_max_chars": 4000,
"vector_store_path": "", // "" = index lives in memory only
"semantic": {
"enabled": true,
"threshold": 0.51,
"top_k": 3,
"chunk_size": 256,
"chunk_overlap": 64
}
},
"codex_modes": [
{
"name": "debug", // required
"model_name": "qwen/Qwen3.8-Flash-Next", // required; "" = pass through
"description": "…", // required; indexed semantically
"examples": ["Napraw błąd w …"], // indexed semantically
"keywords": ["debug", "crash"], // scored: 1.0 (or weights[name])
"phrases": ["root cause:3"], // scored: 2.0 (or the :weight suffix)
"patterns": ["\\broot\\s+cause\\b"],// scored: 3.0
"weights": {"debug": 3} // per-keyword overrides
}
]
}
Required keys: top-level settings and codex_modes; inside settings, trigger_model and fallback_mode; inside
each mode, name, model_name and description. Everything else has a default (see
Defaults at a glance). description and examples are not decoration — with the semantic
layer enabled they are the classifier's training set.
Environment variables#
All overrides use the prefix LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_ and are applied in place at plugin
construction (so after a restart, never mid-session).
| Variable | Purpose |
|---|---|
…_CONFIG |
Custom config: raw JSON string or path to a file |
…_TRIGGER |
Trigger model value (default auto_codex) |
…_MODEL_<MODE> |
Model for one mode, e.g. MODEL_PLAN=… |
…_MODELS |
Per-mode models, e.g. plan=model_a\|test=model_b |
…_MODES |
Pipe-separated whitelist of mode names to keep |
…_FALLBACK_MODE |
Mode used when nothing matches |
…_HEURISTIC_ENABLED |
1/0, true/false, yes/no, on/off |
…_HEURISTIC_MIN_SCORE |
Minimum keyword score to accept a heuristic hit |
…_CLASSIFY_MAX_CHARS |
Character budget of the classified user text |
…_MODEL |
Embedding model for the semantic layer |
…_SEMANTIC_ENABLED |
Toggle the embedding layer |
…_SIMILARITY_THRESHOLD |
Minimum cosine similarity for a semantic hit |
…_TOP_K / …_CHUNK_SIZE / …_CHUNK_OVERLAP |
Embedding router knobs |
…_PERSIST_DIR |
Directory holding index.faiss + docstore.pkl |
…_MODE_<name>_KEYWORDS |
Pipe-separated keyword override for one mode |
Booleans accept 1/0, true/false, yes/no, on/off; numeric values are parsed as int/float and a malformed
value is ignored (with a warning where a logger is present). Unknown mode names in MODELS, MODES, MODEL_<MODE> or
MODE_<name>_KEYWORDS are logged as warnings and otherwise ignored — a typo silently does nothing rather than breaking
routing.
…_MODES filters the mode list but does not move the fallback: if the whitelist excludes the configured
fallback_mode (implement), set …_FALLBACK_MODE too — otherwise validation fails at startup.
Precedence and linting#
…_CONFIG (if non-empty) wins over the bundled JSON; there is no silent fall-through, so a missing file or invalid
JSON raises. Environment variables then win over whatever the file said. At construction the plugin also lints the
signals and warns about settings that can never score, without changing anything:
- a
patternsentry that does not compile (it is dropped by the scorer); - a
patternsentry with upper-case letters (the text is lower-cased before scoring, so it can never match); - a mode with an empty
model_name(requests resolved to it pass through with the trigger model); chunk_overlapnot belowchunk_size(the router clamps it).
Run the router with the logger at INFO/WARNING while changing a config: these warnings are the cheapest way to
discover dead signals.
Configuring the models#
The shipped mapping#
| Mode | Model | Decided by | Why |
|---|---|---|---|
plan |
qwen/Qwen3.8-Flash-Next |
<collaboration_mode> Plan block |
Long-context reasoning, no file edits, latency tolerant |
implement |
qwen/Qwen3.8-Flash-Next |
Fallback for a plain main turn | The hot path: cheap per token, tool-calling reliable |
test |
qwen/Qwen3.8-27B |
Keywords | Reading failures and fixing them rewards a dense model |
git_review |
qwen/Qwen3.8-27B |
Keywords | Diff/git log analysis: precision over speed |
review |
qwen/Qwen3.8-Flash-Next |
Keywords | Mostly reading and describing; short answers |
debug |
qwen/Qwen3.8-Flash-Next |
Keywords | Long traces + hypothesis chains |
aux_title |
qwen/Qwen3.8-27B |
Request class (system thread) | Tiny prompt, structured JSON-schema output |
compaction |
qwen/Qwen3.8-27B |
Request class (request_kind == "compaction") |
Whole-transcript summarization: capacity matters |
The names are what this deployment uses today; the plugin is agnostic to them, it just rewrites strings.
Model facts that matter here#
From the Hugging Face model cards (retrieved 2026-09-24 — re-check for your quantization and serving stack):
| Model | Type | Native context | Main caveat |
|---|---|---|---|
qwen/Qwen3.8-Flash-Next |
MoE, 125 B total / 6 B active | 262 144, up to 1 000 000 | thinking mode on by default |
qwen/Qwen3.8-27B |
dense 27 B | 262 144, up to 1 000 000 | thinking mode on by default |
google/embeddinggemma-300m |
300 M BiEncoder, 768-d output | 2 048 | gated license, no float16 |
Per model:
qwen/Qwen3.8-Flash-Next— MoE with 512 experts (10 routed + 1 shared), 48 layers, hidden 2560, plus a 51 B n-gram embedding table and a 4 B MTP head: cheap per token, which is why it carries the hot modes. The card recommendstemperature=1.0, top_p=0.95, top_k=20for thinking mode.qwen/Qwen3.8-27B— dense, 64 layers, hidden 5120: more expensive per token, more capacity per pass, which is why the transcript-sized auxiliary classes and the failure-analysis modes sit here.- Both chat models expose
enable_thinking,preserve_thinkingandreasoning_effort, and are vision-language models (Codex only ever sends them text). Native context is 262 144 tokens, extensible to 1 000 000 with RoPE scaling — but only if the serving stack is configured for it. google/embeddinggemma-300m— sentence-transformers BiEncoder, 768-d output (MRL-truncatable to 512/256/128), trained on 100+ languages, which is what makes the Polish + Englishexampleslists indexable at all.
Consequences worth internalizing:
- Both chat models default to thinking mode. The plugin forwards
reasoninguntouched, so reasoning depth comes from the CLI (model_reasoning_effort) and the server's chat-template defaults, not from routing config. If a mode feels slow, look there first. - Allocate enough output tokens. Thinking needs room;
model_max_output_tokensthat is too tight truncates reasoning and looks like a broken agent, not like a routing problem. - The embedding model only sees 2 048 tokens. That is why
chunk_sizeis 256 and why over-length queries are windowed and averaged; raisingchunk_sizepast 2 048 buys nothing. - EmbeddingGemma is gated. Either accept the license and
huggingface-cli login, or pointembedding_modelat a local directory (what this deployment does: a path under/models/...instead of the Hub id). Otherwise the semantic layer fails to load at startup and quietly disables itself. - Embeddings run on CPU. A semantic decision costs an embedding pass per request that reaches layer 5; the deterministic layers exist precisely so most turns never pay it.
Hard requirements on the router side#
- A
model_namethe router does not know is a downstream 4xx/5xx, not a routing no-op. Keep the mode→model mapping and the router model list in sync; when you rename a deployment, both places change. - Codex needs tool calling: the provider for every mode must expose it (
tool_calling: truein the model config, and a server started with tool parsing enabled). A mode whose model cannot call tools makes the agent hang or emit text instead ofexec_commandcalls — routing looks "wrong" but is configured correctly. - Context window:
compactionrequests carry the entire transcript. The provider'sinput_size, the server's--max-model-lenand the CLI'smodel_context_windowshould agree; a 256 000-token Codex window against a 32 000 provider is a guaranteed failure on the longest turns. - Mixed models per session are expected. Codex keeps history in the CLI and re-sends it every turn, so switching models between turns is safe; but a mode change mid-session does mean a different tokenizer and different tool-call formatting quirks. Keep modes that must agree on the same model if you observe drift.
- Do not point the trigger at itself.
model_nameequal to the trigger model in a hot mode makes routing invisible and hides misconfiguration; if you want "no routing for this mode", give it the model you actually want to audit.
Choosing which mode gets which model#
Practical order, cheapest first:
- Run with the shipped mapping and read
mode=… source=… model=…from the logs for a day of real work. - Move the modes you actually hit (usually
implement, thentest) to the model you would like to pay less for and compare task success; keep the auxiliary classes (aux_title,compaction) on whichever model summarizes best — they are invisible to you until they are wrong. - If a mode is over-triggered, tune the signals (see Tuning) before you change models; a wrong mode makes any model look bad.
# one mode, one process, no config edit
export LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_MODEL_TEST="qwen/Qwen3.8-Flash-Next"
# several at once
export LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_MODELS="implement=qwen/A|debug=qwen/B"
Tuning and gotchas#
heuristic_min_scoreis the false-positive dial. One plain keyword is1.0, so the default3.0needs a strong keyword ("debug": 3), a phrase (2.0+), or a pattern (3.0). Lower it and generic words start stealing turns; raise it and specialised modes go quiet. Score math is visible in the logs viasimilarity = score/(score+1).planandimplementsignals are dead by design. Their keyword lists exist in the JSON for the embedding layer only. To route planning by text you would have to changeHEURISTIC_MODESin code — prefer Plan Mode in the CLI, or an explicit"agent_mode": "plan"override.- Patterns must be lower-case. Text is lower-cased before scoring;
"\bTODO\b"never matches andlint_signalssays so at startup. weightsonly affects declared keywords. A weight for a keyword that is not inkeywordsis never read.- Phrase weights are inline.
"napraw błąd:3"is a phrase worth 3; a malformed suffix ("napraw błąd:x") is treated as part of the phrase text — a silent no-op. - Keep
classify_max_charsin mind for long first prompts. The budget applies to older messages; the newest one is always classified whole. Raising it gives the keyword and semantic layers more history (and more CPU), lowering it makes the decision depend on the newest message only. - Write Polish signals in both spellings. There is no diacritic folding and no stemming beyond word-start prefixes:
błądnever matchesblad. Every signal users may type without Polish characters needs its ASCII twin — as the shipped lists already do. - Class routing needs Codex metadata. If a proxy or middleware strips
client_metadata(or the JSON string inx-codex-turn-metadata),compactionandaux_titledegrade tomainand get heuristic/fallback routing — the classic "why is my title generation on the coding model" symptom. - A persisted FAISS index is not invalidated when you edit the config. With
PERSIST_DIR/vector_store_pathset,index.faissanddocstore.pklare reloaded as-is. After changing modes, descriptions or examples, delete both files (or point at a new directory) or you keep routing against the old index. - Missing ML extras are silent by intent.
semantic.enabled: truewithoutfaiss/sentence-transformerslogsCodex semantic routing disabled: …and continues deterministically. If you expect layer 5 to work, grep startup logs for that line. - One trigger per plugin.
agentic_routing_codexanswersauto_codex; the semantic plugins answerauto. All of them can be enabled inLLM_ROUTER_UTILS_PLUGINS_PIPELINEat once — they do not interfere as long as the aliases stay distinct. - Trigger matching is exact after
strip()and case-sensitive:" Auto_codex"is not routed. - Routing is per request, not per session. Two turns of one conversation may legitimately land on different models. If you need per-session stickiness, implement it in the caller — this plugin never caches a decision.
Verifying the plugin#
Offline dry run (no network, no embedding model)#
"""Minimal end-to-end check of the Codex routing plugin (deterministic layers only)."""
import json
import os
os.environ["LLM_ROUTER_ROUTING_SEMANTIC_AGENTIC_CODEX_SEMANTIC_ENABLED"] = "false"
from llm_router_plugins.utils.routing.agentic_routing.codex import CodexRoutingPlugin
INSTRUCTIONS = "You are a coding agent running in the Codex CLI."
PLAN = "<collaboration_mode># Plan Mode (Conversational)\n\nThink first.</collaboration_mode>"
DEFAULT = "<collaboration_mode># Collaboration Mode: Default\n\nExecute.</collaboration_mode>"
def turn(text, collaboration=DEFAULT, **turn_meta):
meta = {"request_kind": "turn", "thread_source": "user", "thread_id": "t-1", "turn_id": "u-1"}
meta.update(turn_meta)
items = []
if collaboration:
items.append({"type": "message", "role": "developer",
"content": [{"type": "input_text", "text": INSTRUCTIONS + "\n\n" + collaboration}]})
items.append({"type": "message", "role": "user",
"content": [{"type": "input_text", "text": text}]})
return {
"model": "auto_codex",
"instructions": INSTRUCTIONS,
"input": items,
"tools": [{"type": "function", "name": "exec_command", "parameters": {}}],
"client_metadata": {"thread_id": "t-1", "turn_id": "u-1",
"x-codex-turn-metadata": json.dumps(meta)},
}
plugin = CodexRoutingPlugin(logger=None)
cases = {
"plan mode turn": turn("Zaplanuj migrację bazy danych.", PLAN),
"keyword turn": turn("napraw testy w tests/"),
"plain turn": turn("Dodaj endpoint zwracający status usługi."),
"title generation": turn("Summarize this thread in one line.", None, thread_source="system"),
"compaction": turn("Compact the conversation.", DEFAULT, request_kind="compaction"),
}
for name, payload in cases.items():
out = plugin.apply(payload)
print(f"{name:20} -> model={out['model']:26} agent_mode={out['agent_mode']:11} "
f"source={out['routing']['source']:19} sim={out['routing']['similarity']:.3f} "
f"class={out['routing']['codex_class']}")
Expected output with the shipped config:
plan mode turn -> model=qwen/Qwen3.8-Flash-Next agent_mode=plan source=collaboration_mode sim=1.000 class=main
keyword turn -> model=qwen/Qwen3.8-27B agent_mode=test source=heuristic sim=0.933 class=main
plain turn -> model=qwen/Qwen3.8-Flash-Next agent_mode=implement source=fallback sim=0.000 class=main
title generation -> model=qwen/Qwen3.8-27B agent_mode=aux_title source=class sim=1.000 class=aux_title
compaction -> model=qwen/Qwen3.8-27B agent_mode=compaction source=class sim=1.000 class=compaction
Router smoke test#
curl -sS http://<router-host>:8081/v1/responses \
-H "Authorization: Bearer $LLM_ROUTER_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"auto_codex","input":[{"type":"message","role":"user",
"content":[{"type":"input_text","text":"napraw testy w tests/"}]}]}' \
| head -c 400
Then confirm the router logged mode=test source=heuristic model=qwen/Qwen3.8-27B.
Automated tests#
python -m pytest tests/test_agentic_routing_codex.py -q # plugin, cascade, scoring, config
python -m pytest tests/test_routing_common.py -q # shared routing plumbing
The suite asserts the deterministic layers exactly and exercises the semantic layer through stub routers only, so it
runs without a model or network. agents-conversation/codex/conv-01/ holds the captured real requests (main turn,
title, compaction) the payload builders mirror.
Troubleshooting#
| Symptom | Likely cause | Fix |
|---|---|---|
| Nothing is ever rerouted | plugin not in LLM_ROUTER_UTILS_PLUGINS_PIPELINE, or trigger mismatch |
check the [utils] Registered utility plugin log line, then …_TRIGGER |
Every turn is implement / source=fallback |
signals too narrow, heuristic_min_score too high, semantic off |
lower …_HEURISTIC_MIN_SCORE, add phrases, check …_SEMANTIC_ENABLED |
plan never selected in Plan Mode |
no <collaboration_mode> block reached the plugin (stripped/rewritten) |
verify the developer message survives to the router; "agent_mode": "plan" as override |
| Titles or compaction on the coding model | client_metadata / x-codex-turn-metadata missing or unparsable |
stop stripping metadata; routing.codex_class in the logs tells you what was seen |
| Wrong specialised mode | overlapping keywords; ties go to test → git_review → review → debug |
re-weight, or move the ambiguous phrase to a phrases entry with a weight |
similarity just under 0.51 on good matches |
embedding model / threshold mismatch for your phrasing | lower …_SIMILARITY_THRESHOLD gradually (0.45–0.5 is the usual band) |
model not found after a routing decision |
model_name not declared in the router model config |
add/fix the model in LLM_ROUTER_MODELS_CONFIG |
| Agent stops calling tools on some turns | routed model/provider without tool parsing | tool_calling: true + a server with tool parsing for that model |
| Longest turns fail with context errors | provider input_size < Codex model_context_window |
align them; compaction requests carry the whole transcript |
| Semantic layer never used, no explanation at INFO | extras missing (single Codex semantic routing disabled: … warning) |
pip install ".[ml]" or set …_SEMANTIC_ENABLED=false on purpose |
| Edits to modes/examples change nothing, with a persist dir | stale index.faiss / docstore.pkl reloaded as-is |
delete both files (or change …_PERSIST_DIR) and restart |
Startup fails with CodexRouting: … |
invalid config (trigger, modes, fallback, semantic params) | the message names the offending key or env var |
Reference#
Module map#
| Module | Responsibility |
|---|---|
payload.py |
Codex wire format → immutable CodexRequest; request classes; user-text assembly |
scoring.py |
keyword / phrase / regex scoring, score / (score + 1) confidence mapping |
classifier.py |
the six-layer cascade, HEURISTIC_MODES, CLASS_ROUTED_MODES, RoutingDecision |
semantic.py |
optional cosine-similarity layer, acceptance threshold, fail-open lookups |
config.py |
JSON loading, env overrides, validate_args, lint_signals |
plugin.py |
CodexRoutingPlugin: trigger gate, router construction, payload annotation |
Defaults at a glance#
| Key | Default |
|---|---|
settings.trigger_model |
auto_codex |
settings.fallback_mode |
implement |
settings.heuristic_enabled |
true |
settings.heuristic_min_score |
3.0 |
settings.classify_max_chars |
4000 |
settings.semantic.enabled |
true |
settings.semantic.threshold |
0.51 |
settings.semantic.top_k |
3 |
settings.semantic.chunk_size / chunk_overlap |
256 / 64 |
| keyword / phrase / pattern weight | 1.0 / 2.0 / 3.0 |
| heuristic candidate order | test, git_review, review, debug |
| class-routed modes | compaction, aux_title |
| embedding device | cpu |
MAX_QUERY_WINDOWS |
4 (over-length query windows) |
Reference deployment (example — your hosts will differ)#
| Role | Declaration | Notes |
|---|---|---|
| trigger | auto_codex → api_type: "builtin" |
resolved by this plugin, never sent upstream |
implement, plan, review, debug |
qwen/Qwen3.8-Flash-Next → api_type: "vllm", http://<vllm-host-1>:7000 |
input_size: 256000, tool_calling: true |
test, git_review, aux_title, compaction |
qwen/Qwen3.8-27B → api_type: "vllm", http://<vllm-host-2>:7000 + http://<vllm-host-3>:7000 |
two replicas, balanced by the router |
| embeddings | local directory with google/embeddinggemma-300m |
CPU, no Hub download at startup |
On the CLI side this deployment runs model_context_window = 256000 and model_max_output_tokens = 32000, matching
the 256 000-token providers so that even a compaction request stays inside what the backend accepts. The shipped
embedding_model is a local path precisely because the Hub build is gated: with no cached weights and no
huggingface-cli login, the semantic layer disables itself at startup.
See also#
- Semantic Routing Plugins reference — §3 of the shared routing README, plus its plugin comparison.
llm_router_plugins/utils/routing/embedder.py: the shared BiEncoder + FAISS router used by both plugins.llm_router_plugins/resources/routing/agentic_routing_codex.json: the shipped mode definitions and signal lists.