Agentic knowledge layer (agentic_knowledge)#
A knowledge layer for coding agents that lives in the proxy, with no cooperation from the agent. It reads the work state the routing plugin has already derived (work mode, phase evidence, request class), retrieves what this repository has already learned from the Semantic Search Engine (SSE), injects it into the request, and learns from the traffic itself: verified fixes, dead ends, and the window that a compaction is about to throw away. The retrieval result becomes a routing signal as well — an agent spinning in a loop with no known fix is moved to a stronger model (knowledge-or-escalate).
- Plugin name (registry key):
agentic_knowledge - Class:
llm_router_plugins.utils.knowledge.agentic_knowledge.plugin.AgenticKnowledgePlugin - Default config:
llm_router_plugins/resources/knowledge/agentic_knowledge.json - Env prefix:
LLM_ROUTER_KNOWLEDGE_AGENTIC_ - Worker:
llm-router-agentic-kb-worker - Requires: the SSE client and Redis — both optional, see Install
It is designed to stand behind agentic_routing_codex in the utils pipeline: it reads payload["routing"],
payload["agent_mode"] and payload["model"] as the router left them, and never touches the classifier.
What the plugin changes#
| Key | Value |
|---|---|
input / system |
one extra context block when knowledge was found (adapter-specific) |
model |
the escalation model, only when the escalation policy fired |
routing.knowledge |
annotation of what this plugin did (see Annotation) |
routing.model_before_knowledge |
previous model, only when this plugin replaced it |
Everything else is forwarded unchanged. The plugin never rejects a request.
Fail-open everywhere except the config file. A non-dict payload, an unclaimed payload, a parse error, an SSE
timeout, SSEAPIError, a missing sse_lib, a Redis outage — all return the payload as it came, with at most one
warning that carries no document content. Malformed Redis connection settings disable the features that need Redis and
keep the policy. A malformed policy (unknown key, out-of-range weight, non-positive budget) raises at startup, the
same fail-hard rule the Codex router uses.
Quick start#
1. Install#
pip install -e ".[agentic-knowledge]" # sse-lib>=0.1.0 + redis>=5
sse_lib depends only on requests. Without it, and without a configured host, the plugin loads and does nothing.
Documentation uploads require both the client and SSE server to support the upload_files(file_metadata=...)
contract; update both together. Older clients cannot preserve documentation metadata through file uploads.
2. Load it into the pipeline, after routing#
export LLM_ROUTER_UTILS_PLUGINS_PIPELINE="agentic_routing_codex,agentic_knowledge"
Knowledge needs the routing decision, so it must run after the routing plugin. Registered in
MAIN_UTILS_REGISTRY under agentic_knowledge; instantiated once per process.
3. Point it at SSE and Redis#
export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_HOST="https://sse.internal"
# A pre-issued token, or a login for an engine that authenticates by username and password
# (the client logs in on its first call; a token, when set, wins).
export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_TOKEN="…"
export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_USERNAME="…"
export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_API_PASSWORD="…"
export LLM_ROUTER_KNOWLEDGE_AGENTIC_REDIS_HOST="redis.internal"
No SSE_API_HOST ⇒ no reads (requests pass through). No REDIS_HOST ⇒ no writes, and the generation cache
falls back to one that lives in the process only (reads stay stateless across workers, see Stateless mode).
4. Run the indexing worker#
export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_EMBEDDER="<embedder served by your SSE>"
export LLM_ROUTER_KNOWLEDGE_AGENTIC_SSE_RERANKER="<reranker served by your SSE>"
llm-router-agentic-kb-worker
Without a worker the layer only reads: nothing new is ever learned. The embedder and reranker are required because the worker creates a collection the first time a scope produces knowledge.
The worker is a second process, started by the operator — the router never spawns it. The llm-router checkout
wraps that for a deployment whose configuration lives in the router's own run script:
scripts/plugins/agentic_knowledge/run-agentic-kb-worker.sh reads that script's assignments (never executes
it, so no second server starts), exports them, and runs the worker with the rest of its arguments.
# from the llm-router checkout (sibling of this one)
bash scripts/plugins/agentic_knowledge/run-agentic-kb-worker.sh [ROUTER_SCRIPT] [--once | --no-retain | --log-level …]
Run it with --once from cron to drain a backlog, or without it as a daemon. --once logs what the batch did,
because otherwise a cron run that indexed nothing and a cron run that did nothing are indistinguishable.
5. Verify#
python -m pytest tests -q -k agentic_knowledge
bash scripts/agentic-kb-eval.sh --enable-escalation
The evaluation script replays recorded sessions offline (no SSE, no Redis) and prints a JSON report; see Evaluation.
How it works#
graph LR
P[Payload] --> R[agentic_routing_codex]
R -->|routing, agent_mode| K[agentic_knowledge]
K --> A[AgentAdapter Codex / Claude Code]
A -->|AgentTurn| M[EpisodeMiner]
A -->|AgentTurn| Q[QueryBuilder per mode]
Q --> C[ContextAssembler]
C -->|search| S[SSE]
C --> RC[Redis generation cache]
M -->|XADD| ST[Redis Stream]
ST --> W[Worker CLI]
W --> FM[FastMasker]
W -->|index_documents / upload_files| S
C --> E[EscalationPolicy]
E --> O[Payload + context + routing.knowledge]
One request passes through: adapter selection → parse → mine once → publish once → read (cache → search → assemble) → inject → escalate → annotate. Mining and publishing happen exactly once per request — the write path never re-annotates, so an escalation decision cannot be overwritten by it.
A request the router classified aux_title is returned before any of this — a title generation has no repository
knowledge to use, and none is mined from it, so it carries no annotation.
| Module | Responsibility |
|---|---|
plugin.py |
AgenticKnowledgePlugin — the whole request path, fail-open, annotation |
config.py |
policy JSON + ENV, validation, availability properties |
adapters/base.py |
AgentAdapter (ABC), AgentTurn, KnowledgeEvent, InjectionSpec |
adapters/codex.py |
Codex: reuses codex.payload, codex.phase; full read and write |
adapters/claude_code.py |
Claude Code: scope, latest user text, injection into system; read-only |
scope.py |
repository scope from the declared working directory; collection names |
signature.py |
error/stack trace → stable signature + hash |
query.py |
query and SSE filters per work mode |
assembler.py |
weighted voting over sources, dedup, character budget, rendering |
cache.py |
GenerationContextCache (Redis) + in-process one when Redis is absent |
redisclient.py |
RedisClientHandle: lazy, fork-aware Redis client shared by cache and pub |
miner.py |
EpisodeMiner: recipe / playbook / deadend / loop / compaction → KnowledgeEvent |
docs_ingestor.py |
scans the repository's markdown into doc documents and keeps them current |
doc_context.py |
the standing map of that documentation a session carries, and where it lives |
events.py |
KnowledgeEventPublisher: SET NX dedup + XADD |
escalation.py |
EscalationPolicy.decide |
sse.py |
KnowledgeStore: thin fail-open façade over sse_lib |
documents.py |
document kinds, names, metadata contract |
evaluation.py |
counterfactual replay over recorded sessions |
Scope#
Knowledge is only worth retrieving inside the repository that produced it. The CLI announces no repository, so the
identifier is the one the agent itself prints: <cwd> in Codex's <environment_context>, the Working directory line
in Claude Code's system. A compaction re-sends the whole conversation, and that block then sits behind the
developer instructions — so the leading messages are searched (a fraction of a millisecond, bounded) rather than the
first item assumed. A directory quoted in a tool output is not a declaration: only what the agent sends as a message
names a repository. That path becomes a path-free scope key (scope.strategy):
repo_basename(default) — the name of the checked-out directory;parent_directory—<parent>/<repo>, for monorepos under a common parent;scope.overrides— an explicitdeclared path → scope keytable, for checkouts that share a basename.
One collection per scope keeps isolation out of the query: a search of one repository cannot return a document of
another, even if a metadata filter is forgotten. The absolute path itself is never indexed. With no declared directory
the scope is unknown, and both reads and writes are skipped (scope: "" in the annotation).
Collections and documents#
| Kind | Category (engine) | Produced by |
|---|---|---|
recipe |
recipe |
test_failure → patch → test pass |
playbook |
playbook |
patch → test pass with no failure in between — the correct steps |
deadend |
deadend |
a repeated failure, a dead end |
doc |
doc |
the repository's markdown — Documentation |
session_archive |
session_archive |
a request classified compaction |
One collection per repository, <prefix>_<scope>: every kind above is
written into it, told apart by its engine category rather than by the name
of the collection. The separation is as hard as a collection was — a search
names the categories it asks for, and the engine resolves them to a set of
document names before the vector leg scores anything
(semantic.py: documents_names_from_categories → search_in_documents), so
a request that did not ask for the documentation never sees it, and its
max_results is spent within what survived the cut. One embedder, one
chunking and one min_similarity already served all three collections, so the
layout gives up nothing the reader used, and it leaves behind two collections
per repository to create, prune and monitor.
"collection_layout": "split" keeps the older three-collections spelling —
<prefix>_<scope> for mined knowledge, <prefix>_<scope>_sessions for the
archive, <prefix>_<scope>_docs for the documentation — for a deployment that
was indexed with it: it changes only the names, and a search restricted to one
collection and a search restricted to the matching categories of one collection
return the same documents. The layout is a property of the data at rest, so a
deployment switching it switches both the reader and the writer at once.
Default <prefix> is agentic_kb. The engine's character set — letters,
digits, _, . — is enforced at configuration time, on both the prefix and
the scope, because the scope is a checkout's directory name and the engine
refuses a hyphen (a 500) in a collection name: llm-router becomes
<prefix>_llm_router. A document is named <kind>:<YYYYMMDD>:<fingerprint> —
the day bucket is what retention ages on. Every entry is one document with its
metadata in metadata_json, filtered by eq/in.
Read path: retrieval per work mode#
QueryBuilder builds a query from what the mode is actually doing:
| Mode | Query |
|---|---|
debug |
signature of the last test_failure + the latest user text |
test |
the test command + touched paths |
implement |
latest user text + touched paths |
plan |
latest user text |
git_review, review |
paths from patches |
* (fallback) |
the general composition: user text + paths + failure, for a mode routing did not resolve |
Each mode has a policy — the sources it reads, their weights, max_items, and a character budget:
"debug": {
"max_chars": 6000,
"sources": [
{"category": "recipe", "weight": 1.0, "max_items": 3},
{"category": "deadend", "weight": 0.9, "max_items": 3},
{"category": "playbook", "weight": 0.5, "max_items": 2},
{"category": "session_archive", "weight": 0.6, "max_items": 3},
{"category": "doc", "weight": 0.4, "max_items": 2, "doc_types": ["guide", "note"]}
],
"escalation_model_name": "qwen/Qwen3.8-27B"
}
A mode with no policy of its own — routing resolved none, or a name this policy
does not know — falls back to the one named *, so the request is asked of the
base rather than served empty. An explicit policy always wins; a configuration
without a * entry behaves as before (such a request is mined, but not read).
The shipped policy reads every kind the miner can write — including
playbook, where the mode is about doing rather than un-breaking — and reads
documentation in the modes that previously read none of it: debug asks for the
guide and the notes about the module the failure sits in, test for the guide
and the contributing rules the change has to keep, and implement for the
dead ends it would otherwise walk back into.
A source may also restrict the phases it is read in — the work the episode was
doing (test, implement, …), which is not the document kind: a recipe mined
while implementing is still a recipe — or its modes, the routing mode the
router resolved for the request that produced it. An empty list — the default —
reads the source in every phase or mode. The restriction becomes engine-side
metadata_filters on phase (or mode), one filter per name, which the engine
unions; it intersects with the source's categories. Phases and modes share the
single metadata slot the engine allows, so a policy may restrict one dimension
but not both in one request (configuration error), an archived source never
carries either — its thread_id filter owns the slot — and one source without a
restriction leaves the whole request unrestricted (sources of one collection are
retrieved by one search).
The doc source carries a third dimension of the same kind: doc_types,
the documentation kinds it reads — readme, guide, changelog,
contributing, security, license, agent, note. One eq filter per
kind, the engine unions them. It is the doc source's own restriction, in its
own collection and its own request, so it never collides with the phase and
mode dimensions above; a doc source that also lists phases or modes is a
configuration error, as is a doc_types list on any other source (a mined
episode carries no doc_type, and the filter would read as empty). The
shipped policy is deliberate per work mode: implement reads readme, guide
and contributing, plan adds changelog and note, and review reads
guide, contributing and security. agent (an AGENTS.md) is in no
list: the agent already receives that file in its own context, and retrieving
it again would only spend the mode's budget on words it has been given.
{"category": "recipe", "weight": 1.0, "max_items": 3, "phases": ["test", "debug"]}
{"category": "deadend", "weight": 0.9, "modes": ["debug"]}
{"category": "doc", "weight": 0.8, "max_items": 3, "doc_types": ["guide", "contributing", "security"]}
ContextAssembler normalizes SSE's score_normalized, multiplies by the source weight, adds a bonus when a document's
error signature or paths match the current turn, and a further documentation bonus when a retrieved guide
documents a file the turn is touching — docs/export.md for an edited src/export.py — a link no retrieval leg
can see, deduplicates by document name, and cuts to the budget. A doc hit whose declared kind the mode does not read
is dropped on the client, as the second voice of the engine's own doc_type filter. Sections are
rendered in a fixed order inside constant markers:
<agentic_knowledge>
## Verified fixes in this repo
## Verified procedures in this repo
## Known dead ends (do not repeat)
## Exact context from before compaction
## Conventions
</agentic_knowledge>
Every entry is cited with the phase it was learned in — [recipe · in test · 2026-10-12 · src/a.py],
[playbook · in test · 2026-10-12 · src/a.py] — so the agent can tell a fix it verified in a test run from one it
learned by editing. A procedure is never the "strong recipe" that holds a looping model in place: it carries no
failure signature, and what was never broken is not evidence about what is.
A repository whose first episode has not been indexed yet has no collection at all, and the engine says so with an error code rather than an empty list; the read path answers it as an empty one (nothing known yet, no warning per request) until the worker creates the collection on that first event.
The block is injected as the last element of input (Codex: a developer message) — not into instructions, so
the prompt cache prefix of the CLI survives and the block stays removable. The client does not echo our element back,
so the same block is re-injected for the whole command generation from the cache. If <agentic_knowledge> is already
in the payload the injection is skipped, which keeps a second pass through the pipeline idempotent. Empty result ⇒ no
element at all.
Generation cache#
GenerationContextCache stores the injected set under <key_prefix>:ctx:<scope>:<generation> (generation =
the Codex turn_id, the same identity the Codex memory treats as one command generation; when the CLI sends none,
the adapter falls back to a hash of thread_id + input items, then to root_turn_id) with cache_ttl_seconds
(default 3600). A hit
returns byte-identical text and issues zero SSE calls. The entry is refreshed when the mode changes or the phase
evidence fingerprint moved. The escalated model is stored with the entry, so a loop escalated earlier in a generation
stays escalated even when a later request no longer repeats the loop by itself.
Write path#
EpisodeMiner reads the turn's phase evidence — collect_phase_evidence from the Codex package, no second parser —
and emits recipe for a test_failure → patch → test pass episode, playbook for a patch → test pass episode
that no failure opened, and deadend for a failure that repeats with an identical signature. A turn that went right
from the first try writes no recipe — there was no failure to file it under — and before the playbook kind it
wrote nothing at all: the correct steps of most tasks never reached the base. A playbook is therefore read back
where the question is how this repository does a thing that has not gone wrong yet, not where a failure is the
question: the body leads with what was asked for, then what was changed, then the command that proved the change.
A stretch that a verified recipe already covers (the recipe keeps every patch that followed its failure, so the
last, wider test run is usually inside its diff) is not written twice. A stretch a test run never accepted is not
written either: an intention indexed as a verified procedure is a guess the next session would trust. Each event carries the phase its own evidence names — the mode most of its items are in,
the newest one breaking a tie, the turn's mode when it names none — which the worker writes into the document
header and into metadata_json.phase, verbatim (it is a filter key, like thread_id). Codex resends the whole history with every request, so episodes are recomputed each time and
deduplicated before publishing: SET NX <fingerprint> EX <dedup_ttl_seconds> in front of XADD, and XADD … MAXLEN ~
<max_stream_len>. If XADD fails the dedup key is released, so the episode is retried rather than silently lost.
A request whose repository is unknown publishes nothing: its documents would have no repository's collection to live
in — the reader asks for a scope's collection, never the bare prefix — and every scope-less checkout would archive
into that one prefix, which is the isolation the scope exists to prevent.
A green command verifies a recipe only when its full test scope can be established as equivalent to the failing
command. A subset, selection filter, unknown option, or shell composition is conservatively rejected; command
comparison is not truncated. This may omit valid fixes using unsupported runners/options rather than certify them
without evidence. Each failed attempt owns its patches and resulting failure: in failure → patch 1 → failure →
patch 2 → pass, the dead end contains only patch 1, while the recipe retains the cumulative patch history.
The worker (llm-router-agentic-kb-worker) consumes the stream with a consumer group, masks every document with
FastMasker and discards the mappings (no re-identification is possible downstream) — the filter keys
(thread_id, scope, kind, mode, phase, …) are written verbatim so a document can be read back — creates the
collection on first use, then sends mined events through index_documents and documentation through upload_files,
preserving category and options metadata in both paths. Logs never contain document text. Retention
deletes only this plugin's own documents older than the configured TTL (session_archive on archive.ttl_seconds).
Events with an unknown version are acknowledged and logged; an indexing error is retried up to a bounded number of
attempts and then dropped with a log, so one poison event cannot stall the stream.
Masking is fail closed for writes: an exception or an invalid masker result for the body, signature, or masked
metadata prevents indexing the entire document and follows that same bounded retry path. There is no raw-text
fallback or new quarantine queue; after the retry limit the event is dropped. The user's request is not blocked.
Compaction archive#
A request classified compaction is written, never read from: the full window is chunked at message/tool-result
boundaries (archive.max_chunk_chars) and indexed into the repository's collection with category
session_archive and the metadata thread_id, window_id and position. The next main request of the same
thread reads that category filtered by thread_id, so the exact lines a summary dropped come back as
## Exact context from before compaction. The filter is what keeps the archive to one thread: the mined knowledge in
the same collection carries a thread_id of its own, so a request that asked for both categories and one thread
would silently narrow the knowledge to that thread — which is why the builder never does.
Documentation (doc)#
The doc category is the repository's own documentation: README, CONTRIBUTING, everything under docs/. It is the
cheapest project-specific knowledge there is, it is already written, and the miner cannot produce it — so the layer
reads the directory the request declared and publishes it.
llm-router-agentic-kb-docs --path /repo/webapp --dry-run # what would be published
llm-router-agentic-kb-docs --path /repo/webapp # publish it to the stream
| Step | What happens |
|---|---|
| Scan | docs.patterns below repo_root, skipping docs.excluded_dirs and never .git, symlinks or deep trees |
| Publish | one doc event per whole file, without local splitting, to the same stream the miner uses |
| Upload | the worker masks the snapshot and sends it as a UTF-8 .txt attachment through sse_lib.SSEClient.upload_files; files for one collection share a request, including on retry |
| Index | SSE splits each uploaded document using indexing_options identical to collection chunking; file_metadata carries category=doc and options, including source_file, doc_type and heading |
A file is republished when its content changes, not when it is looked at: the digest of the whole file feeds its
document fingerprint and the scan state (Redis, one key per scope: <key_prefix>:docs:<scope>) compares digests, so
touch and a branch checkout across ten unchanged files publish nothing. A file whose content changed gets a new
document name — the digest is part of it — and the scan then publishes a docs_prune control message asking the worker
to retire the name it superseded, which is also what removes the documents of a file that was deleted. A scan that
hit max_files or max_total_bytes publishes no prune at all: pruning against a partial listing would delete the
documentation the scan never reached, and the outcome says truncated: true.
Switching to uploads does not automatically rewrite already-indexed, unchanged files: the scan still compares
content digests. Their existing documents remain searchable; a content change publishes the replacement through
the new upload path and prunes the old names. docs.max_section_chars and DOCS_MAX_SECTION_CHARS no longer control
indexing — chunk sizes are configured on the worker instead.
The documentation lives apart from mined knowledge, and the state says so: a listing records the collection it was written for, and a policy that moves the documentation to another collection finds no listing at all and republishes the repository where it now belongs, instead of believing files are indexed in a collection nobody searches.
The scan runs on the request path and is bounded by docs.rescan_interval_seconds per process (a repository is looked
at once per interval, not once per request) and by the byte and file ceilings. It reads the filesystem as the router
sees it: where the router does not run next to the agents, repo_root is not a directory it can open, the scan
reports skipped: no-such-directory, and documentation is published with the command above from a machine that can.
docs.auto_index: false keeps the category and the retrieval but takes the filesystem away from the request path: only
llm-router-agentic-kb-docs (or the CLI's own --dry-run) publishes then.
The documentation request is not the knowledge request. Mined knowledge is asked with the mode's query — the failure
signature in debug, the test command in test — because that is what a recipe is filed under; documentation is prose
about the code, filed under the module and the concept, so it is searched with the user's own words plus the touched
paths and their stems. The failure signature never rides on a documentation search, and the citation says where the
quoted section came from: [doc · guide · docs/export.md › Export helpers].
Standing documentation context (doc_context)#
Retrieval answers a question. A digest answers the question the agent has before it has one: what this project is, which documentation it keeps, and what the pages cover. It is injected on every request a mode reads documentation in, whether or not this turn's query matched anything, so a long session stops re-learning the same shape of a repository command after command.
## Documentation in this repository
This repository documents itself; the map below is what it has written down, and any of it can be
asked for in full.
- README.md (readme): Demo Service; Install; Deploy
- docs/guide.md (guide): Guide; Export helpers; Rollback
| Property | What it means |
|---|---|
| Structural | paths, kinds and headings — no prose. The write path masks a document before indexing it; a digest of README sentences built on the request path would carry them unmasked to a model. A heading is what the layer already files unmasked, and it is enough to aim a question. |
| Stable | Kept per repository, not per request or generation. It changes when the documentation's outline changes; rewording a paragraph moves its fingerprint, not its text, so the prompt under the provider's cache does not move either. |
| Budgeted first | Charged against modes.<mode>.max_chars before anything is retrieved; a digest that will not fit the mode is not injected at all rather than squeezing the block. |
| Cheap | Built from the listing the scan already made, and read from Redis (<key_prefix>:docs:context:<scope>, TTL docs.context_ttl_seconds) by every other request of the session — one GET, no filesystem walk per command. |
Two levels, by design: the map (always present, docs.context_max_chars) and the deep
chunks (what the query matches, in the repository's collection under category doc). The annotation reports which was
injected:
"doc_context": {"level": "digest", "chars": 187, "source_files": ["README.md", "docs/guide.md"]}
The digest is refreshed by the scan that already runs (docs.rescan_interval_seconds) and by
llm-router-agentic-kb-docs, so a repository whose router runs on another machine still gets its
map — the command writes it while it publishes. docs.context_enabled: false returns the layer to
retrieval alone.
Linking knowledge to documentation#
A mined fix and a written rule about the same module are two halves of one
answer, and neither search connects them: a recipe is filed under the failure it
fixed, a page under the module it explains. The map the request already carries
resolves the link — a recipe that names src/export.py is rendered with a
pointer to docs/export.md, the page that shares the file's stem:
## Verified fixes in this repo
- [recipe · 2026-10-16 · src/export.py] flush the queue before closing the writer. → docs/export.md
The pointer is resolved on the reader side, against the documentation map of
the scope, so it works whether or not this turn's query retrieved the page — no
extra search, no extra request. A documentation entry points at nothing (it is
the pointed-at), and a mined entry whose files no page documents gets no tail:
a citation is worth its characters only when it leads somewhere. The link
follows the same rule the stem bonus does: a named page (docs/export.md)
matches by file stem (src/export.py), a generic page (README.md,
AGENTS.md) documents the directory it sits in and matches only inside it — a
root README.md is not documentation of everything, and a page an entry
already cites is not pointed at a second time. The tail is charged against
modes.<mode>.max_chars like the rest of the line.
Two deliberate limits. The link is not stored in the document's metadata:
the engine does not return document metadata on a search hit, so a stored
related_docs would be write-only — resolved where it is read instead of where
nobody can read it. And a documentation page never counts toward
best_recipe_score: a README is not a verified fix, and documentation must
not push a mode past its escalation threshold.
Knowledge-or-escalate#
EscalationPolicy.decide(...) considers escalation only when enabled and all of these hold:
routing_source != "explicit"— the operator who named a model is never overridden;- the mode has an
escalation_model_name; loop_count >= loop_threshold(default3) — the same error signature failed that many times in the generation;- no eligible recipe trial remains for this failure.
Retrieval relevance (best_recipe_score) is not recipe reliability. To earn a trial, a recipe must actually
be injected, match the current signature_hash, declare a nonempty test_command, and meet strong_recipe_score
(default 0.75). This is a minimum verification-provenance check, not a count of independent confirmations;
legacy documents lacking these fields remain readable but cannot hold back escalation.
The generation cache records the failure count at the first qualifying injection, even before the loop threshold.
After the threshold is reached, a higher count means recipe_did_not_resolve_loop and permits escalation regardless
of newly retrieved recipes. Replaying the same evidence does not consume the trial. This detects failure after
exposure, not proof that the agent applied the advice. Without exposure history (including a cache outage),
a recipe can keep the model only at loop_threshold; beyond it, recipe_trial_limit prevents indefinite suppression.
Outcomes are none, keep (a qualifying recipe still has its trial) and escalate, with reasons disabled,
explicit_mode, no_loop, no_escalation_model, strong_recipe, loop_without_recipe,
recipe_did_not_resolve_loop, recipe_trial_limit. On escalate the adapter
sets the model and routing.model_before_knowledge records the previous one. The policy is off in the shipped
config (escalation.enabled=false) — enable it only after the replay shows the loop detection is right, because a
flaky test looks exactly like a loop. The "already tried" list comes from the current turn's dead ends plus retrieved
deadend hits and lands in ## Known dead ends (do not repeat).
Annotation#
payload["routing"]["knowledge"] — safe for logs: no document text, no tokens, no hosts, no full session ids.
{
"plugin": "agentic_knowledge",
"adapter": "codex",
"scope": "webapp",
"injected": 3,
"chars": 1180,
"sources": {"recipe": 2, "deadend": 1},
"cache": "hit", // hit | miss | disabled
"loop_count": 3,
"escalation": {"action": "escalate", "reason": "loop_without_recipe",
"from": "auto_codex", "to": "qwen/Qwen3.8-27B"},
"write": {"published": 1, "deduplicated": 0, "errors": 0, "status": "ok"}, // ok | partial | unavailable | disabled
"docs": {"status": "ok", "files": 12, "published": 1, "pruned": 1},
"doc_context": {"level": "digest", "chars": 187,
"source_files": ["README.md", "docs/guide.md"]}
// the standing documentation map injected for the session;
// absent when docs.context_enabled is off or nothing is scanned
// absent when documentation is off or no repository is known;
// status ok | skipped | disabled | unavailable, `reason` when it
// did not run, `truncated: true` when a ceiling cut the scan
}
Configuration#
Policy is JSON (reviewable, versionable, no secrets); every connection is ENV only. Precedence: ENV → JSON →
built-in defaults. A value present but malformed is an error, except connection settings, where a missing host just
means the feature is unavailable. read.timeout_seconds is the budget a whole request may spend on the engine, and a
search starts only while a whole call — SSE_API_TIMEOUT — still fits before it ends. It is therefore a sum of
calls, not a per-call number: the shipped policy asks up to three questions of the repository's collection per mode
and budgets three calls of the
two-second default. A budget that equals one call is raised at validation by a small margin, because the deadline is
read from the clock a moment before a call is made and a budget with no room over it would start no search of any
request. When the budget cannot hold every call of a mode, the later ones are not started and the plugin says so once
per repository, naming both numbers.
{
"adapters": ["codex", "claude_code"],
"collection_prefix": "agentic_kb",
"scope": {"strategy": "repo_basename", "overrides": {}, "default": ""},
"read": {"enabled": true, "timeout_seconds": 1.5, "min_similarity": 0.35,
"hybrid_search": true, "rerank_results": false,
"cache_ttl_seconds": 3600, "max_lookups_per_generation": 20},
"modes": { "debug": {…}, "test": {…}, "implement": {…},
"plan": {…}, "git_review": {…}, "review": {…} },
"claude_code": {"mode_map": {}},
"write": {"enabled": true, "stream_key": "agentic_kb:events",
"max_stream_len": 100000, "dedup_ttl_seconds": 604800,
"max_patch_chars": 4000,
"archive": {"enabled": true, "max_chunk_chars": 2000,
"ttl_seconds": 1209600}},
"escalation": {"enabled": false, "loop_threshold": 3, "strong_recipe_score": 0.75},
"docs": {"enabled": true, "auto_index": true,
"patterns": ["*.md", "*.markdown", "*.mdx"],
"excluded_dirs": ["node_modules", "dist", "build", "vendor"],
"max_files": 200, "max_file_bytes": 200000,
"max_total_bytes": 2000000,
"rescan_interval_seconds": 21600, "state_ttl_seconds": 7776000,
"context_enabled": true, "context_max_chars": 700,
"context_ttl_seconds": 86400}
}
The retrieval legs of one search are read.hybrid_search and read.rerank_results,
switched by …_HYBRID_SEARCH and …_RERANK_RESULTS. They also decide the scale every score comes
back on — the engine's results.score_scale: a squashed cross-encoder logit with the reranker on,
a Reciprocal Rank Fusion rank in a plain hybrid search, and a cosine similarity when the legs are
off — so escalation.strong_recipe_score is a threshold on that scale: a reranker logit under
rerank_results: true, an RRF rank otherwise. Raising it against an RRF scale is a no-op in
practice, which is why the reranker stays off by default: it costs a model call per search and a
collection whose engine serves no reranker answers nothing with it on.
How one request reads the engine#
A mode asks one question per collection it reads — mined knowledge, the session archive, the repository's documentation — and the questions are asked one after another. Two numbers bound what that costs, and they are not the same number:
| Knob | Means |
|---|---|
…_SSE_API_TIMEOUT |
what one call to the engine may take |
…_READ_TIMEOUT_SECONDS |
what one request may spend on the engine in total |
A search is started only while a whole call still fits before the budget runs out, so
…_READ_TIMEOUT_SECONDS is the ceiling a request really waits rather than a per-call figure
multiplied by the number of sources. It is never allowed below the per-call timeout — a budget in
which no call fits would read nothing and report an empty knowledge base. When a budget is spent
mid-request the remaining sources are skipped and one warning names the scope and how many were
lost; raise the budget or shorten the mode's sources. Measured against a real engine on another
host, a hybrid search takes a median of ~1.7 s, which is why a 1.5 s per-call ceiling lost every
search it made.
The plugin also asks the engine not to guess the language of the query
(…_SSE_IGNORE_LANGUAGE_DETECT, on by default). The engine otherwise detects the language of the
question and applies it twice: as a metadata filter on the vector leg (text_language == <lang>)
and as the name of the PostgreSQL full-text configuration (pl → polish). Neither fits a
knowledge base, which is indexed in the language of the code and asked in the language of the
operator — the first filters the answers away, and a cluster without that configuration answers the
full-text leg with an HTTP 500 rather than with nothing. With the guess off the engine drops the
filter and uses simple, which exists everywhere.
A mode accepts max_chars, sources, escalation_model_name and cheap_model_name. The last one is parsed and
kept, but no policy in the code reads it yet — it is the model a strong recipe would recommend — so setting it
changes nothing: routing stays as the router left it.
Environment variables#
All use the prefix LLM_ROUTER_KNOWLEDGE_AGENTIC_. Booleans accept 1/0, true/false, yes/no, on/off.
| Variable | Default | Purpose |
|---|---|---|
…_ENABLED |
true |
master switch for the whole plugin |
…_CONFIG |
bundled JSON | raw JSON string or path to a policy file |
…_READ_ENABLED |
true |
injection on/off (writes may stay on) |
…_WRITE_ENABLED |
true |
publishing on/off (reads may stay on) |
…_ARCHIVE_ENABLED |
true |
compaction archiving on/off |
…_ESCALATION_ENABLED |
false |
model escalation on/off |
…_COLLECTION_PREFIX |
agentic_kb |
prefix of every SSE collection (letters/digits/_/. only) |
…_KEY_PREFIX |
agentic-kb |
Redis key namespace |
…_SSE_API_HOST |
empty | empty ⇒ no reads |
…_SSE_API_PREFIX |
empty | empty ⇒ sse_lib.endpoints.DEFAULT_API_PREFIX (api); set it only when the engine is mounted elsewhere |
…_SSE_API_TOKEN |
empty | SSE token (never logged); when set, it wins over a login |
…_SSE_API_USERNAME / …_SSE_API_PASSWORD |
empty | engine login for an engine that authenticates by login rather than by a token; the client logs in on its first call; neither is ever logged |
…_SSE_API_TIMEOUT |
2.0 |
ceiling of one call to the engine, clamped to 0.1…10 s |
…_SSE_VERIFY |
true |
TLS certificate verification |
…_READ_TIMEOUT_SECONDS |
4.0 |
read budget: what one request may spend on the engine in total; never smaller than …_SSE_API_TIMEOUT, so one call always fits |
…_SSE_IGNORE_LANGUAGE_DETECT |
true |
ask the engine not to guess the language of the query (it would filter the vector leg to it and name the full-text configuration after it) |
…_MIN_SIMILARITY |
0.35 |
floor for a hit |
…_HYBRID_SEARCH |
true |
fuse the vector leg with the full-text leg |
…_RERANK_RESULTS |
false |
re-score the hits with the collection's cross-encoder |
…_CACHE_TTL_SECONDS |
3600 |
lifetime of one generation's context |
…_MAX_LOOKUPS_PER_GENERATION |
20 |
per-process ceiling on searches, applied with or without Redis |
…_MAX_STREAM_LEN |
100000 |
XADD MAXLEN ~ |
…_DEDUP_TTL_SECONDS |
604800 |
episode dedup window |
…_MAX_PATCH_CHARS |
4000 |
patch text admitted into an event |
…_ARCHIVE_MAX_CHUNK_CHARS |
2000 |
archive chunk size |
…_ARCHIVE_TTL_SECONDS |
1209600 |
archive retention |
…_DOCS_ENABLED |
true |
repository documentation is part of the knowledge base |
…_DOCS_AUTO_INDEX |
true |
a request may trigger a scan; false ⇒ only the CLI publishes |
…_DOCS_MAX_FILES |
200 |
files read per scan; a scan that hits it prunes nothing |
…_DOCS_MAX_FILE_BYTES |
200000 |
largest documentation file read |
…_DOCS_MAX_TOTAL_BYTES |
2000000 |
bytes read per scan |
…_DOCS_RESCAN_INTERVAL_SECONDS |
21600 |
per-process interval between scans of one repository |
…_DOCS_STATE_TTL_SECONDS |
7776000 |
lifetime of a scope's scan state (90 days) |
…_DOCS_CONTEXT_ENABLED |
true |
the session carries the standing documentation map |
…_DOCS_CONTEXT_MAX_CHARS |
700 |
ceiling of the map injected into a request |
…_DOCS_CONTEXT_TTL_SECONDS |
86400 |
how long a scope's map survives without a refresh |
…_LOOP_THRESHOLD |
3 |
repetitions of one signature that make a loop |
…_STRONG_RECIPE_SCORE |
0.75 |
a recipe this good keeps the routed model |
…_REDIS_HOST |
empty | empty ⇒ no writes, no cache |
…_REDIS_PORT / …_REDIS_DB / …_REDIS_PROTOCOL |
6379 / 0 / 3 |
Redis connection |
…_REDIS_USERNAME / …_REDIS_PASSWORD |
empty | optional ACL / AUTH |
…_REDIS_SSL, …_REDIS_SSL_* |
off | TLS and its material (same contract as Codex memory) |
…_REDIS_SOCKET_CONNECT_TIMEOUT / …_REDIS_SOCKET_TIMEOUT |
1.0 |
short positive timeouts |
Redis settings come from these fixed variables only — never from JSON, a Redis URL, REDIS_*, LLM_ROUTER_REDIS_* or
AUTH_REDIS_*. They are read through connection_from_env(prefix) reused from codex/state.py.
Worker environment#
| Variable | Default | Purpose |
|---|---|---|
…_WORKER_GROUP |
agentic-kb |
consumer group name |
…_WORKER_CONSUMER |
worker-<ts>-<pid> |
this consumer's name in the group |
…_WORKER_BATCH |
32 |
events claimed per read |
…_WORKER_BLOCK_MS |
5000 |
how long a read waits for work; the worker's own socket timeout is raised above this automatically, so an idle block does not surface as a TimeoutError |
…_MAX_PENDING |
1000 |
pending length above which the worker warns |
…_SSE_EMBEDDER |
empty | required to create a collection |
…_SSE_RERANKER |
empty | required to create a collection |
…_SSE_INDEX_TYPE |
HNSW |
HNSW or IVF_FLAT |
…_SSE_CHUNK_TOKENS |
400 |
chunk size sent on collection creation and every indexing request; 0 leaves the engine default (500 tokens, embedded text cut at 508) |
…_SSE_CHUNK_OVERLAP |
60 |
chunk overlap in tokens, must be < SSE_CHUNK_TOKENS |
…_RETENTION_INTERVAL_SECONDS |
21600 |
seconds between retention passes; 0 disables |
The same max_tokens_in_chunk and number_of_overlap_tokens are sent when a collection is created and on every
indexing request. Changing …_SSE_CHUNK_TOKENS/…_SSE_CHUNK_OVERLAP does not rewrite an existing collection or its
documents: recreate and re-index the affected collection to keep it consistent. The
default 400/60 sits under the 508 tokens the embedder cuts at, which a chunk of the engine's 500 default does not.
CLI (worker): --config <path>, --once (drain one batch and exit — cron friendly), --no-retain, --clean-redis-memory-cache, --log-level.
--clean-redis-memory-cache drops the layer's cached state in Redis before the worker starts: assembled generations, the scan state of the documentation (and its standing digest), and the deduplication claims that say which events were already published. Use it after you delete a collection — the scan state still says the documentation is indexed and the claims still say its events were seen, so nothing is written again until every TTL expires on its own, and the clean turns the next scan into a real re-index. The event stream, its consumer group and the per-entry attempt counters are deliberately left alone: they are a queue and its bookkeeping, and emptying them would lose episodes that were published and not yet indexed. A clean that fails stops the run with a log rather than indexing on a cache that was not cleared.
The worker logs its configuration block (engine, stream, group, consumer, chunking, retention) on start and then:
a per-batch summary with read/indexed/dropped counts and elapsed time, per-event details at DEBUG
(kind, scope, thread, fingerprint), collection creation, retention deletions, docs pruning, and an
idle heartbeat every five minutes. The level comes from --log-level, else from
LLM_ROUTER_IN_DEBUG (on → DEBUG), else from LLM_ROUTER_LOG_LEVEL, else INFO — the last two
being what the router's run script already exports, so a debug deployment gets a debug worker
without repeating the switch. Failures are logged with the engine's own message, not just the
exception class, and credentials are reported by their kind (auth=token), never by their value.
Documentation command#
llm-router-agentic-kb-docs --path /repo/webapp --dry-run # list, publish nothing
llm-router-agentic-kb-docs --path /repo/webapp --json # publish, machine-readable outcome
llm-router-agentic-kb-docs --path /repo/webapp --no-prune # publish, keep superseded documents
llm-router-agentic-kb-docs --path /repo/webapp --force # replace queued docs and republish all files
Same policy, same scan, same stream: the command reads files and publishes events, so the worker masks and indexes
what it indexes for a mined episode and there is no second write path. It therefore needs the stream
(…_REDIS_HOST) and never the engine. --scope overrides the scope key the proxy would derive from the directory, so
a repository indexed by hand lands in the collection its agents read from.
--force is for rebuilding after deleting the SSE database or collection, even when file contents have not changed.
It deletes existing doc and docs_prune entries for the selected scope from the configured Redis stream,
clears their deduplication claims (including claims for current files whose old entries were already trimmed),
and republishes every scanned file. It refreshes the scan state after successful publication. Other scopes,
recipes and session archives are untouched; it does not delete the SSE collection itself. --no-prune
still prevents retiring superseded documents in SSE, but does not disable replacement of queued events.
Stop the indexing workers and pause other publishers for this scope before using --force, then restart them
afterwards. Stream deletion cannot cancel a batch a worker has already fetched. Replacement is not atomic:
if Redis fails midway, the command exits unsuccessfully; rerun with --force to replace the partial attempt.
Forced scans bypass docs.max_files, docs.max_file_bytes and docs.max_total_bytes, so a large changelog
does not block rebuilding. File patterns, excluded directories and symlink restrictions still apply; normal scans
keep their configured limits. Forced scans read all selected files into memory, so preview large repositories first.
Unreadable files/directories or a truncated scan refuse replacement before touching the stream.
--force --dry-run previews the same unlimited scan and never accesses Redis.
The CLI reads Redis settings from exported environment variables, not from the scan directory or an automatically
loaded .env file. --path . only chooses what to scan; previous indexing does not persist connection settings.
An error saying LLM_ROUTER_KNOWLEDGE_AGENTIC_REDIS_HOST is unset occurs before inspecting any indexed state,
and --force cannot fix it. Export the same LLM_ROUTER_KNOWLEDGE_AGENTIC_REDIS_HOST (and port, DB and credentials,
if applicable) used by the worker in this shell before rerunning. --config selects the knowledge policy,
not Redis connection settings.
Adapters#
Codex (full)#
Claims requests with the Codex Responses shape, reuses CodexPayloadParser, collect_phase_evidence and
CodexPhaseConfig rather than parsing again. Full read and write: request class, activity, phase evidence, touched
paths and the compaction window all come from it. Contract tests for CodexRequest/PhaseEvidence live in
tests/test_agentic_knowledge_codex_adapter.py — a change there must keep those contracts.
Tool output is recognized by the Codex envelope (Process exited with code N); without it the phase pipeline cannot
see a test failure, so no test_failure evidence, no recipe, no loop.
A patch is read out of the envelope, never out of the tool name: releases differ in how they carry one, and a
current CLI applies it by piping *** Begin Patch … *** End Patch to apply_patch as a here-document inside
exec_command. Both shapes count — the patch/input argument of an apply_patch call and a here-document the
command applies — and a quoted envelope in a command that applies nothing does not. Without this a turn edits files
invisibly, EpisodeMiner.verified() stays false, and the collections stay empty however healthy the layer looks.
Claude Code (basic)#
Claims requests whose routing.plugin is agentic_routing_claude_code, or that carry the Anthropic messages +
system shape. Scope comes from the Working directory line in system, latest_user_text from the last user
message, session identity from metadata.user_id when present. The work mode is the tier from routing.mode mapped
through claude_code.mode_map — empty by default, which means every request is served as implement. The block is
appended to system (string or list of blocks; cache_control is left alone). set_model writes payload["model"].
No miner and no escalation: the Anthropic payload carries no phase evidence, so nothing can be verified and a loop cannot be seen. Claude Code requests read the shared knowledge base; they do not add to it.
Privacy, masking, retention#
- The worker masks every document with
FastMaskerand drops the mappings, so an indexed document cannot be reversed. - Paths outside
repo_rootare dropped from metadata; only paths relative to the repository are stored. Absolute paths and the declared working directory are never indexed. - Annotation and logs carry no document content, no tokens, no hosts, no full session ids.
- Retention deletes only documents of this plugin's own collections, and only of the kind it names.
session_archiveages onarchive.ttl_seconds(default 14 days), which is the shortest-lived and largest-volume kind. - Masking is rule-based, not semantic. Fixtures with production data are not safe here: do not enable the layer on repositories whose test data carries real customer content.
Stateless mode#
With no Redis the plugin still reads: no writes at all, and a cache that lives in this process only
(InMemoryContextCache, the same contract and the same TTL as the Redis one), so a single-worker deployment loses
nothing but the sharing. max_lookups_per_generation is enforced by an in-process LookupLimiter in both modes, so
the ceiling on SSE calls is per worker either way and grows with the number of workers — there is no shared
guarantee. The escalated model carries over to the next request of the generation only inside the worker that decided
it. There is deliberately no local file fallback, because a fallback that diverges between workers is worse than no
fallback at all.
Evaluation#
bash scripts/agentic-kb-eval.sh # no KB, read
bash scripts/agentic-kb-eval.sh --enable-escalation
evaluation.py replays recorded sequences (tests/data/agentic_knowledge_eval.json, anonymized prefixes from
agents-conversation/) offline. For request k the fake store is fed only events mined before k, so no future
knowledge leaks backwards, and the question is whether retrieval would have returned the fix the session eventually
used. Metrics: recipe_hit_rate, turns_potentially_saved, missed_saves, loop_detection_precision/recall,
escalations, injected_chars_per_turn, reported per variant: no_kb, read, read_escalate. Use it to decide
whether escalation is worth enabling; tests/test_agentic_knowledge_evaluation.py keeps the replay deterministic and
leak-free.
Limitations#
- Signature quality decides everything. Loop detection and recipe relevance both hang on
signature.py, which normalizes paths, line numbers, addresses, hashes and timestamps. Verify on real output before trusting a threshold. - Token cost. Re-injecting every request of a generation costs tokens, bounded per mode by
max_chars; thecharsfield of the annotation is the meter. - Document count. Every knowledge entry is a separate SSE document, so collections grow; hence retention and
max_stream_len. - False escalations. A flaky test looks like a loop. Identical signatures and
loop_thresholdhelp; keepESCALATION_ENABLEDoff until the replay is convincing. - No architectural decisions. Documentation is copied into the knowledge base, never distilled from a session: extracting the decision behind a change would need generation, and a wrong summary indexed as a convention is worse than the source file an agent can read itself.
- Documentation is read where the router runs. The scan reads
repo_rootfrom this host; a router that does not run next to the agents cannot scan their repositories and its outcomes sayskipped: no-such-directory. Publish the documentation withllm-router-agentic-kb-docsfrom a machine that can see them. - A lost scan state republishes a repository's documentation.
<key_prefix>:docs:<scope>is what keeps a scan from re-publishing unchanged files and what names the documents a change supersedes; flush it and the next scan publishes everything again, leaving the previous copies of each file in the collection until they are deleted by hand. The listing names the collection it was written for, so changingcollection_prefix— or moving the documentation into<prefix>_<scope>_docs— republishes it once where it now belongs and leaves the old collection alone. - The standing context is a map, not a summary. It says which documentation exists and what
the pages are titled; it does not know what they argue. A repository whose README is one long
sentence and whose pages are titled
Notesgets a map that says little — the fix is better headings, which helps the retrieval leg too. - A
doc_typeslist excludes what it does not name. A guide filed asnotebecause its file says nothing else is read only by a mode that listsnote(or lists nothing); a mode that names kinds reads exactly those. When a repository's prose stops appearing, read the mode'sdoc_typesbefore suspecting the retrieval. The reverse also holds: delete documents out of the engine by hand and the scan will not republish them — their content is unchanged and their fingerprints are deduplicated — so delete the scope's state key together with them. - Coupling to the
codexpackage. The adapter imports its parser, phase and state modules explicitly; their contracts are pinned by tests.
See also#
llm_router_plugins/utils/routing/agentic_routing/codex/README.md— the routing plugin this layer builds on..plans/in_progress/agentic-knowledge-layer-re.md— the plan this implementation follows.scripts/agentic-kb-eval.sh— the offline replay.