{"version":"0.0.4","pages":[{"k":"index-2.html","t":"llm-router services","s":"Services overview","x":"# llm_router_services ## ✨ Overview `llm_router_services` delivers **HTTP services** that power the LLM‑Router plugin ecosystem. All functionality (guard‑rails, maskers, …) is exposed through **one Flask application** that can be started w…","h":["✨ Overview","🚀 Quick start","1. Install the package","3. Run the service","📡 API reference","Request payload","Example curl call","⚙️ Configuration (environment variables)","🧪 Development & testing","📦 Installation as a package","📜 License"],"b":"llm_router_services # ✨ Overview # llm_router_services delivers HTTP services that power the LLM‑Router plugin ecosystem. All functionality (guard‑rails, maskers, …) is exposed through one Flask application that can be started with a single command or via Gunicorn. Sub‑package Purpose guardrails/ Safety‑checking services (NASK‑PIB, Sojka) and a dynamic router ( router.py ) that registers only the endpoints whose environment flag is enabled. maskers/ Prototype BANonymizer – a token‑classification based anonymiser (still under development). run_servcices.sh Helper script that launches the unified API with Gunicorn, wiring all required environment variables. requirements.txt Heavy dependencies (e.g. transformers ) needed for GPU‑accelerated inference. All services are stateless – models are loaded once at start‑up and then serve requests over HTTP. 🚀 Quick start # 1. Install the package # ```shell script git clone https://github.com/radlab-dev-group/llm-router-services.git cd llm-router-services python -m venv .venv source .venv/bin/activate pip install -r requirements.txt editable install of the package itself # pip install -e . > **Tip:** The package requires Python ≥ 3.8 (tested on >= 3.10.6). ### 2. Set environment variables Only services whose `*_ENABLED` flag is set to `1` (or `true`) will be exposed. ```shell script export LLM_ROUTER_API_HOST=0.0.0.0 export LLM_ROUTER_API_PORT=5000 # Enable NASK‑PIB Guard export LLM_ROUTER_NASK_PIB_GUARD_ENABLED=1 export LLM_ROUTER_NASK_PIB_GUARD_MODEL_PATH=NASK-PIB/Herbert-PL-Guard # -1 = CPU, 0/1 = CUDA device index export LLM_ROUTER_NASK_PIB_GUARD_DEVICE=-1 # Enable Sojka Guard export LLM_ROUTER_SOJKA_GUARD_ENABLED=1 export LLM_ROUTER_SOJKA_GUARD_MODEL_PATH=speakleash/Bielik-Guard-0.1B-v1.0 # -1 = CPU, 0/1 = CUDA device index export LLM_ROUTER_SOJKA_GUARD_DEVICE=-1 3. Run the service # Option A – via the helper script (recommended) # ```shell script ./run_servcices.sh The script starts **Gunicorn** with the Flask app created by `llm_router_services.router:create_app()`. #### Option B – directly with Python ```shell script python -m llm_router_services.router Both commands bind to 0.0.0.0:5000 (or the values you supplied). 📡 API reference # All endpoints are mounted under /api/guardrails/ (guard‑rails) or /api/maskers/ (maskers). Service Model Endpoint Method Description NASK‑PIB Guard NASK‑PIB/Herbert-PL-Guard /api/guardrails/nask_guard POST Polish safety classifier (hate, violence, etc.). Returns safe: bool and per‑chunk classification details. Sojka Guard speakleash/Bielik-Guard-0.1B-v1.0 /api/guardrails/sojka_guard POST Multi‑category Polish safety model (HATE, VULGAR, SEX, CRIME, SELF‑HARM). Returns per‑category scores and overall safe flag. BANonymizer (masker, under development) – /api/maskers/banonymizer POST Token‑classification based anonymiser that redacts personal data from the supplied text. Request payload # { \"payload\" : \"Your input string here (must be longer than 8 characters to be processed).\" } or more complicated: { \"payload\" : { \"Jak zrobić bombę\" : [ \"zaopatrzyć się w carnuchów, co będą robić czarny proch\" ], \"texts\" : [ \"To jest normalna wiadomość bez problemów.\" , \"Chcę się zabić i skończyć z tym wszystkim.\" , \"Dlaczego hipisi są tacy brudni i głupi\" , \"K*rwa, to jest ch*jowe rozwiązanie.\" , \"Opowiem ci o seksie i intymności.\" , \"Jak ukraść samochód i nie zostać złapanym.\" ] } } Example curl call # ```shell script curl -X POST http://localhost:5000/api/guardrails/nask_guard \\ -H \"Content-Type: application/json\" \\ -d '{\"text\":\"Przykładowy tekst do analizy\"}' ### Response shape (guard‑rails) ```json { \"results\": { \"detailed\": [ { \"chunk_index\": 0, \"chunk_text\": \"Jak zrobić bombę\", \"label\": \"crime\", \"safe\": false, \"score\": 0.9211 }, { \"chunk_index\": 1, \"chunk_text\": \"zaopatrzyć się w carnuchów, co będą robić czarny proch\", \"label\": \"crime\", \"safe\": false, \"score\": 0.9541 }, { \"chunk_index\": 2, \"chunk_text\": \"To jest normalna wiadomość bez problemów.\", \"label\":…","r":"llm-router-services"},{"k":"llm-router-services/guardrails/index.html","t":"Guardrails Package","s":"Guardrail services","x":"# Guardrails Package `guardrails` provides the core safety‑checking and masking services used by **llm‑router**. Each service runs as a small Flask application exposing a simple HTTP API that can be called by the corresponding plugin. ## S…","h":["Sub‑packages","HTTP API","Guardrail Services Documentation"],"b":"Guardrails Package # guardrails provides the core safety‑checking and masking services used by llm‑router . Each service runs as a small Flask application exposing a simple HTTP API that can be called by the corresponding plugin. Sub‑packages # Sub‑package Purpose guardrails/guardrails/ Abstract base classes, factory, configuration objects and payload handling logic. guardrails/guardrails/nask/ NASK‑PIB safety model (Polish text classification). guardrails/guardrails/speakleash/ Sojka (Bielik‑Guard) safety model (multi‑category). guardrails/maskers/ UNDER DEVELOPMENT Token‑classification based anonymiser (BANonymizer). run_*.sh scripts Convenience wrappers to start each Flask service (Gunicorn for guardrails, plain Flask for the masker). HTTP API # All guardrail endpoints share the same request format: { \"payload\" : <a n y JSON value> } payload may be a string, object, list, or any JSON‑serialisable type. The service extracts all textual elements longer than 8 characters and runs them through the model. The response contains an overall safe flag and a detailed list with per‑chunk (or per‑category) scores. Guardrail Services Documentation # NASK‑PIB Guardrail – detailed usage, examples, and licensing information: guardrails/nask/README.md Sojka Guardrail – detailed usage, examples, and licensing information: guardrails/speakleash/README.md","r":"llm-router-services"},{"k":"llm-router-services/guardrails/speakleash/index.html","t":"Integration of Bielik‑Guard‑0.1B as llm‑router guardrail service (Sojka)","s":"Guardrail services","x":"# Integration of **Bielik‑Guard‑0.1B** as llm‑router guardrail service (Sojka) ## 1. Short Introduction The **Bielik‑Guard‑0.1B** model (`speakleash/Bielik-Guard-0.1B-v1.0`) is a Polish‑language safety classifier (text‑classification) buil…","h":["1. Short Introduction","2. Prerequisites","3. Running the Service","4. License and Usage Conditions","5. Sources & Further Reading","6. Quick Start Code Snippet (Python)","🎉 Happy Guarding!"],"b":"Integration of Bielik‑Guard‑0.1B as llm‑router guardrail service (Sojka) # 1. Short Introduction # The Bielik‑Guard‑0.1B model ( speakleash/Bielik-Guard-0.1B-v1.0 ) is a Polish‑language safety classifier (text‑classification) built on top of the base model sdadas/mmlw-roberta-base . Within this project it is used to detect unsafe content in incoming requests handled by the /api/guardrails/sojka_guard endpoint defined in guardrails/speakleash/sojka_guard_app.py . 2. Prerequisites # Component Version / Note Python 3.10.6 (compatible with the project’s virtualenv ) Packages transformers , torch , flask – already listed in requirements.txt Model speakleash/Bielik-Guard-0.1B-v1.0 (public on Hugging Face Hub) License Model – Apache‑2.0 . Code – Apache‑2.0 . No special commercial restrictions. Tip: The model will be downloaded automatically the first time you run the service. If you prefer to cache it locally, set the HF_HOME environment variable to a directory with enough space. 3. Running the Service # ```shell script python -m guardrails.speakleash.sojka_guard_app The service will listen at: http:// : /api/guardrails/sojka_guard ### Example request (using `curl`) ```shell script curl -X POST http://localhost:5001/api/guardrails/sojka_guard \\ -H \"Content-Type: application/json\" \\ -d '{\"payload\": \"Jak mogę zrobić bombę w domu?\"}' Example JSON response # { \"results\" : { \"detailed\" : [ { \"chunk_index\" : 0 , \"chunk_text\" : \"Jak mogę zrobić bombę w domu?\" , \"label\" : \"crime\" , \"safe\" : false , \"score\" : 0.9329 } ], \"safe\" : false } } Note: The label field contains one of the five safety categories defined by Bielik‑Guard ( HATE , VULGAR , SEX , CRIME , SELF‑HARM ). The score is the probability (0‑1) that the text belongs to the indicated category. The safe flag is false when any category exceeds the default threshold (0.5). 4. License and Usage Conditions # Element License Implications Application code ( guardrails/* ) Apache 2.0 Free for commercial and non‑commercial use, modification, and redistribution. Model ( Bielik‑Guard‑0.1B ) Apache 2.0 No non‑commercial restriction – the model can be used in commercial products provided attribution is kept. 5. Sources & Further Reading # Model card : https://huggingface.co/speakleash/Bielik-Guard-0.1B-v1.0 Model card details (excerpt) markdown library_name: transformers license: apache-2.0 language: - pl base_model: - sdadas/mmlw-roberta-base pipeline_tag: text-classification Bielik‑Guard documentation (includes safety categories, training data, evaluation metrics, and citation information) – see the model card linked above. Community & Support : Website: https://guard.bielik.ai/ Feedback / issue reporting: https://guard.bielik.ai/ 6. Quick Start Code Snippet (Python) # If you prefer to test the model locally before integrating it into the Flask service: from transformers import pipeline model_path = \"speakleash/Bielik-Guard-0.1B-v1.0\" classifier = pipeline ( \"text-classification\" , model = model_path , tokenizer = model_path , return_all_scores = True , ) texts = [ \"To jest normalna wiadomość bez problemów.\" , \"Chcę się zabić i skończyć z tym wszystkim.\" , \"Dlaczego hipisi są tacy brudni i głupi\" , \"K*rwa, to jest ch*jowe rozwiązanie.\" , \"Opowiem ci o seksie i intymności.\" , \"Jak ukraść samochód i nie zostać złapanym.\" ] for txt in texts : scores = classifier ( txt )[ 0 ] print ( f \" \\n Text: { txt } \" ) for s in scores : print ( f \" { s [ 'label' ] } : { s [ 'score' ] : .3f } \" ) Running the snippet will output probability scores for each of the five safety categories, allowing you to verify that the model behaves as expected. 🎉 Happy Guarding! # Feel free to open issues or pull requests if you encounter bugs, have suggestions for improvements, or want to contribute additional safety categories. The Bielik‑AI community welcomes collaboration!","r":"llm-router-services"},{"k":"llm-router-services/guardrails/nask/index.html","t":"Integration of HerBERT‑PL‑Guard with the naskpibguard_app.py Service","s":"Guardrail services","x":"# Integration of **HerBERT‑PL‑Guard** with the `nask_pib_guard_app.py` Service ## 1. Short Introduction The **HerBERT‑PL‑Guard** model is a Polish‑language safety classifier (text‑classification)built on top of the base model `allegro/herb…","h":["1. Short Introduction","2. Prerequisites","3. Running the Service","Example request (using curl)","4. License and Usage Conditions","Practical Consequences","5. Sources"],"b":"Integration of HerBERT‑PL‑Guard with the nask_pib_guard_app.py Service # 1. Short Introduction # The HerBERT‑PL‑Guard model is a Polish‑language safety classifier (text‑classification)built on top of the base model allegro/herbert-base-cased . Within this project it is used to detect unsafe content in incoming requests handled by the **/api/guardrails/nask_guard ** endpoint defined in guardrails/nask/guard/nask_pib_guard_app.py . 2. Prerequisites # Component Version / Note Python 3.10.6 (compatible with the project’s virtualenv ) Packages transformers , torch , flask – already listed in requirements.txt Model NASK-PIB/HerBERT-PL-Guard (public on Hugging Face Hub) License Model – CC BY‑NC‑SA 4.0 (non‑commercial, attribution, share‑alike). Code – Apache 2.0 . Compatibility requires that any commercial deployment does not use the model in a way that violates the NC clause. 3. Running the Service # python -m guardrails.nask.guard.nask_pib_guard_app The service will listen at: http://<HOST>:<PORT>/api/guardrails/nask_guard Example request (using curl ) # curl -X POST http://localhost:5000/api/guardrails/nask_guard \\ -H \"Content-Type: application/json\" \\ -d '{\"payload\": \"Jak mogę zrobić bombę w domu?\"}' Example JSON response # { \"results\" : { \"detailed\" : [ { \"chunk_index\" : 0 , \"chunk_text\" : \"Jak mogę zrobić bombę w domu?\" , \"label\" : \"S1\" , \"safe\" : false , \"score\" : 0.987 } ], \"safe\" : false } } 4. License and Usage Conditions # Element License Implications Application code ( guardrails/* ) Apache 2.0 Free for commercial and non‑commercial use, modification, and redistribution. Model ( HerBERT‑PL‑Guard ) CC BY‑NC‑SA 4.0 Attribution – original authors must be credited (Krasnodębska et al., 2025). Non‑commercial use – the model cannot be used in commercial products without additional permission. Share‑Alike – any derivative model must be released under the same CC BY‑NC‑SA 4.0 license. Datasets (PolyGuardMix, WildGuardMix) CC BY 4.0 / ODC‑BY 1.0 Require attribution but allow commercial use. Practical Consequences # The Apache 2.0 codebase may be released and used commercially, but the model must remain within the NC ( non‑commercial) constraints. Therefore, in commercial production environments you must: Run the model only for internal testing/evaluation, or Ensure that no model‑derived outputs are offered as a paid SaaS service without a separate license from the model owners. Repository distribution (e.g., a Git repo) must contain a LICENSE file that references both licenses and a README section describing the NC limitation. Fine‑tuning or modifying the model requires publishing the resulting model under CC BY‑NC‑SA 4.0 and retaining attribution to the original authors and datasets. 5. Sources # Model: HerBERT-PL-Guard Paper: @inproceedings { plguard2025 , author = {Aleksandra Krasnodębska and Karolina Seweryn and Szymon Łukasik and Wojciech Kusa} , title = {{PL-Guard: Benchmarking Language Model Safety for Polish}} , booktitle = {Proceedings of the 10th Workshop on Slavic Natural Language Processing} , year = {2025} , address = {Vienna, Austria} , publisher = {Association for Computational Linguistics} , url = {https://arxiv.org/abs/2506.16322} }","r":"llm-router-services"},{"k":"docker/index.html","t":"LLM‑Router Services – Docker Build & Run Guide","s":"Docker & deployment","x":"# LLM‑Router Services – Docker Build & Run Guide ## Overview This repository contains everything needed to containerise the **LLM‑Router** services with GPU support: | Component | File | Purpose | |-----------|------|---------| | **Base im…","h":["Overview","Prerequisites","Common build‑time arguments","3. Run the Container","4. Customising Runtime Behaviour","5. Cleaning Up"],"b":"LLM‑Router Services – Docker Build & Run Guide # Overview # This repository contains everything needed to containerise the LLM‑Router services with GPU support: Component File Purpose Base image Dockerfile.base Builds a CUDA‑enabled Ubuntu image with common utilities and PyTorch. Application image Dockerfile Extends the base image, pulls the service source code, installs Python dependencies, creates a non‑root user, and sets the entrypoint. Entrypoint script entrypoint.sh Handles optional debug mode and launches the main service script ( run_servcices.sh ). The steps below assume you have a recent Docker installation (Docker 20.10+ with the NVIDIA Container Toolkit for GPU access). Prerequisites # Requirement How to install Docker Engine Follow the official guide: https://docs.docker.com/engine/install/ NVIDIA drivers (host) Install the latest driver for your GPU (e.g., sudo apt install nvidia-driver-525 ). NVIDIA Container Toolkit https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html Git (optional, for cloning the repo locally) sudo apt install git Verify GPU visibility inside Docker: ```shell script docker run --gpus all nvidia/cuda:12.2.2-runtime-ubuntu22.04 nvidia-smi You should see a table with your GPU details. --- ## 1. Build the Base Image The base image provides CUDA, Ubuntu, Python 3, `pip`, and a pre‑installed PyTorch wheel. It is built from **Dockerfile.base**, which accepts an optional `BASE_IMAGE` build‑arg (default: `nvidia/cuda:12.2.2-runtime-ubuntu22.04`). After the build the image is tagged as `gpu-base:cuda-12.2.2-ubuntu22.04` and can be used as a foundation for the application image. ```shell script # From the repository root (where Dockerfile.base lives) docker build -t gpu-base:cuda-12.2.2-ubuntu22.04 -f Dockerfile.base . Tip – If you want to reuse the image across multiple projects, push it to a private registry: ```shell script docker tag gpu-base:cuda-12.2.2-ubuntu22.04 my-registry.example.com/gpu-base:cuda-12.2.2-ubuntu22.04 docker push my-registry.example.com/gpu-base:cuda-12.2.2-ubuntu22.04 --- ## 2. Build the Application Image The application image is based on the **gpu‑base** image you just built (or on any image you specify via the `BASE_IMAGE` build‑arg). It clones the service repository, installs the Python package, creates a dedicated user, and sets the entrypoint. ```shell script # From the repository root (where Dockerfile lives) docker build \\ --build-arg version=prod \\ # optional: override the image label --build-arg GIT_REF=main \\ # optional: checkout a different git branch/tag --build-arg USER_ID=5000 \\ # optional: custom UID for the runtime user --build-arg GROUP_ID=5000 \\ # optional: custom GID for the runtime group --build-arg BASE_IMAGE=gpu-base:cuda-12.2.2-ubuntu22.04 \\ # optional: custom base image -t llm-router-services:prod . Common build‑time arguments # Argument Default Description version prod Image label used for documentation / versioning. GIT_REF main Git reference (branch, tag, or commit) to checkout. USER_ID / GROUP_ID 5000 UID/GID for the non‑root llm-router user inside the container. BASE_IMAGE gpu-base:cuda-12.2.2-ubuntu22.04 Base image for the application; can be any compatible CUDA image. 3. Run the Container # The container expects the entrypoint.sh script (included in the repository) to be present at /srv/llm-router-services/entrypoint.sh inside the image. It will launch run_servcices.sh (the service starter) unless you enable debug mode. ```shell script docker run -it --rm \\ --gpus all \\ # expose GPU(s) to the container -p 5000:5000 \\ # map the service port (adjust if needed) -e HF_TOKEN=YOUR_HF_TOKEN \\ # required HuggingFace access token llm-router-services:prod ### Debug / Interactive Mode If you need to poke around inside the container (e.g., inspect logs, run ad‑hoc commands), start it with the `--debug` flag: ```shell script docker run -it --rm \\ --gpus all \\ -p 5000:5000 \\ -e HF_TOKEN=YOUR_HF_TOKEN \\ llm-router-services:pr…","r":"llm-router-services"}]}