llm-router/docs

Bi-encoder semantic routing guide#

semantic_biencoder_routing is the embedding-based routing plugin. It encodes the user's last statement with a bi-encoder, searches a persisted FAISS index of the targets, and routes to the closest one.

What it does#

SemanticBiEncoderRoutingPlugin activates when payload["model"] == "auto". It builds or loads a FAISS index over the configured targets, embeds the query, and picks the nearest target's model. The index is persisted so it is built once and loaded afterwards. The decision is annotated under payload["routing"].

It shares the common routing layer with the other embedding router: the EmbeddingRouter in llm_router_plugins/utils/routing/embedder.py, the RoutingTarget in target.py, and the config helpers in common.py.

Configuration#

Environment variables under LLM_ROUTER_ROUTING_SEMANTIC_BIENCODER_: the targets, the embedder model, the similarity threshold, and the index persistence directory. See llm_router_plugins/utils/routing/semantic_biencoder/README.md.

When to use it#

Use semantic_biencoder_routing when the routing decision needs semantic similarity rather than word patterns, and the set of targets is stable enough to index once.