Integration Examples with LLM Router#

This directory contains example boilerplates that demonstrate how easy it is to integrate popular LLM libraries with the router by simply switching the host.

Available Examples#

LlamaIndex – Integration with LlamaIndex (GPT Index)
Using LlamaIndex with Local Models – Guide for mapping OpenAI model names to local models via the router.
LangChain – Integration with LangChain
OpenAI SDK – Direct integration with the OpenAI Python SDK
LiteLLM – Integration with LiteLLM
Haystack – Integration with Haystack

Core Principle#

All examples work on the same principle: just change base_url / api_base to the address of your router, and the router will automatically:

✅ Distribute traffic among available providers
✅ Perform load balancing
✅ Provide health checking
✅ Supply monitoring and metrics
✅ Handle streaming and non‑streaming responses

Quick Start#

Each example can be run directly:

```shell script

LlamaIndex#

python examples/llamaindex_example.py

LangChain#

python examples/langchain_example.py

OpenAI SDK#

python examples/openai_example.py

LiteLLM#

python examples/litellm_example.py

Haystack#

python examples/haystack_example.py ```

Example Structure#

Each example includes:

Basic configuration – how to point the library at the router
Streaming – handling streaming responses
Non‑streaming – handling full responses
Error handling – managing errors

Full Stack with Local Models#

The quick‑start guides for running the full stack with local models are included in the repository:

Gemma 3 12B‑IT – README
Bielik 11B‑v2.3‑Instruct – README

These guides walk you through:

Installing vLLM and the respective model.
Setting up LLM‑Router with the provided models-config.json.
Testing the end‑to‑end flow (router → vLLM).

Follow the linked README files for step‑by‑step instructions to launch a complete stack locally.

Additional Information#

Learn more about the router: