Routerly

Self-hosted LLM gateway that routes requests across AI providers (OpenAI, Anthropic, Gemini, Mistral, Ollama) using intelligent multi-policy scoring — including an LLM-native routing policy. Drop-in compatible: just swap the base URL. No database required, built-in cost tracking, budget enforcement and multi-tenant iso

Automatically checked · GitHub checked 2026-11-14

Routerly

Overview

Routerly

Routerly

One gateway. Any AI model. Total control.

The self-hosted decision and control layer for your LLM providers: composable routing policies with explainable decisions, governance (roles, budgets, audit), and wire-format transparency.
Compatible with the OpenAI and Anthropic APIs: change the base URL, nothing else.

https://www.routerly.ai/

Node.js ≥ 20 License: AGPL-3.0 Self-hosted Docker supported Docker Hub OpenAI compatible Anthropic compatible CLA assistant Last commit GitHub stars GitHub contributors


Quick Start

macOS / Linux:

curl -fsSL https://www.routerly.ai/install.sh | bash -s -- --version v0.5.3

Windows (PowerShell):

powershell -c "& ([scriptblock]::Create((irm https://www.routerly.ai/install.ps1))) -Version v0.5.3"

The commands pin v0.5.3 because the default channel and the Docker latest tag still resolve to an older release. See Installation options.

Docker:

docker run -d \
  --name routerly \
  -p 3000:3000 \
  -v routerly_data:/data \
  -e ROUTERLY_HOME=/data \
  --restart unless-stopped \
  inebrio/routerly:v0.5.3

Then:

  1. Open http://localhost:3000/dashboard and create the admin account (first-run Setup screen).
  2. Log in from the CLI, register a model, create a router and attach the model:
routerly auth login --url http://localhost:3000 --email [email protected]
routerly model add --id gpt-5-mini --provider openai --api-key sk-YOUR_KEY
routerly router create --name "My App"        # prints the router token (sk-rt-…), shown once
routerly router model add "My App" gpt-5-mini
  1. Point your client at http://localhost:3000/v1 (OpenAI format) or http://localhost:3000 (Anthropic format) and use the router token as the API key.

If the service is not running (for example after a source build), start it with routerly start.

Learn more: Quick Start tutorial — Installation options and update channels — Dashboard guide — CLI reference


What Routerly is

Routerly sits between your applications and your LLM providers. It is the layer that decides where each request goes and controls who may send what, while staying invisible to the client.

  • Decide: composable routing policies score every candidate model, and every decision can be explained (routing trace, Playground, per-call usage records). Routing is one function of the layer, not the whole of it.
  • Control: roles and permissions, per-router and per-token budgets, guardrails, PII scrubbing, audit log, notifications.
  • Stay transparent: clients change only the base URL. Routerly adds no custom headers and no non-standard fields, and returns the provider's response as it received it. The only payload changes are the ones you switch on yourself (PII scrubbing, guardrail steering, token optimizers), each off unless configured.

No database and no Redis: configuration lives in JSON files.


Capabilities

  • Routers, orchestrators and passthrough. A router scores models with routing policies. An orchestrator forwards to a priority-ordered list of other routers. A passthrough router forwards a client's own credential straight to OpenAI or Anthropic (optionally alongside real target models).
  • 11 routing policies, combined into one ranking (table below), plus routing profiles: reusable presets for routing, optimizers and security. Four built-in routing profiles (auto, cheap, fast, coding), clonable.
  • Budgets and cost tracking. Every request is priced per token; hard limits (hourly, daily, weekly, monthly, yearly and rolling windows) at global, router and token level block overspending before the call.
  • Guardrails and PII scrubbing. Rule-based and judge-based content guardrails, and PII policies, on requests, responses, or both.
  • 8 token optimizers (session-dedup, ccr, rtk, headroom, json-table, relevance, caveman, llmlingua-2), off by default, enabled per router.
  • Experiments. A/B test whole routers against each other (model sets, profiles, optimizer pipelines, guardrails) without any client change.
  • Resilience. Automatic fallback to the next candidate on provider errors or timeouts.
  • Roles and permissions. Custom roles over 26 granular permissions, per-user two-factor authentication, router members.
  • Audit log of management actions.
  • Notifications to the in-app inbox, email (SMTP, SES, SendGrid, Azure, Google) and webhooks (generic, Slack, Teams, PagerDuty, Discord).
  • MCP server. Personal MCP tokens let Claude Code, Cursor, Cline and other MCP clients inspect and adjust Routerly with your own permissions.
  • Dashboard, CLI and API. Every function is available in all three; the dashboard also has a Playground with routing trace and cost breakdown.
  • Updates. Built-in update check against GitHub Releases; channels latest, current, next, develop; update with routerly update run or from the dashboard.

Routing policies

Policies run in order; each filters or scores the candidates, and the highest combined score wins. A failed call falls back to the next candidate.

PolicyWhat it does
llmAsks a language model to pick the best candidate given the request
semantic-intentClassifies the request intent with embeddings and prefers models suited to it
model-preferencePrefers the model the client named; abstains when the client sends the virtual model routerly/ada
cheapestMinimises estimated cost
healthDeprioritises models with recent errors
performanceFavours models with lower observed latency
capabilityMatches models to task requirements (vision, tools, JSON mode, thinking)
contextFilters models by context window versus prompt size
budget-remainingExcludes models that would push a router over its budget
rate-limitSteers traffic away from rate-limited providers
fairnessBalances load across candidates

How Routerly compares

Last verified: 2026-10-21, against each project's public README and documentation at that date. "n/v" means not verifiable from public documentation.

Routerly (0.5.x)LiteLLM proxyBifrostPortkey OSS gatewayOpenRouter
Self-hosted✅✅✅✅❌ hosted service
Open sourceAGPL-3.0open core, enterprise features under a commercial licenseApache-2.0 (enterprise tier exists)MIT (hosted/enterprise tier exists)n/a
OpenAI-compatible API✅✅✅✅✅
Anthropic Messages API (/v1/messages)✅✅✅ (/anthropic)n/vn/v
Database for core usenone (JSON files)PostgreSQL recommended for production; virtual keys and spend tracking use itn/vn/vn/a
Routing11 composable policies, profiles, orchestratorsfallback, weighted, latency, cost, usage, least-busy strategiesrouting rules, key/provider load balancing, fallbacksfallbacks, retries, weighted load balancingn/v
LLM- or intent-based routing✅ (llm, semantic-intent)n/vn/vn/vn/v
Budgets / spend control✅ global, router, token✅ spend tracking and limits per key/team/user✅ virtual keys, teams, customersn/vn/v
Guardrails✅ built in (rules, judge, PII)✅enterprise tier✅ input/output checks (PII redaction listed as enterprise)n/v
Token optimizers✅ 8, opt-inn/vn/vn/vn/v
MCPmanagement MCP server (not a tool gateway)MCP gatewayMCP client and serverMCP gateway (separate product)n/v
Roles and permissions✅ custom roles, 26 permissionsn/v (SSO listed as enterprise)enterprise tiern/vn/v
SSO (OIDC/LDAP)not yetenterpriseenterprisen/vn/v
Web UI✅ dashboard✅ admin UI✅ built-inlocal log consolehosted dashboard

These gateways overlap with Routerly and are not mutually exclusive: several can sit behind Routerly as an upstream (anything that speaks the OpenAI or Anthropic API can be registered as a custom provider), and Routerly can sit in front of them.


Use Cases

SaaS with multiple tenants

Create one router per tenant, give each its own token and monthly spend cap. A tenant cannot exceed its budget or see another's traffic, and your app code does not change.

Local-first development

Develop against Ollama locally and promote to a hosted model in production. Same API surface in every environment; only the router token changes.

Cost optimisation without quality loss

Combine cheapest, capability and context (or the built-in cheap profile) to send simple requests to cheap models and fall back to a stronger model only when the task needs it. Token optimizers reduce the prompt on top.

Resilience and failover

Register equivalent models across several providers and enable health. Routerly routes around failing endpoints and tries the next candidate on error.

Governance for teams

Roles and permissions decide who can change what, the audit log records changes, guardrails and PII policies apply to every request, and notifications report budget and provider events.

Comparing setups safely

Run an experiment between two routers (for example a cheaper model set against the current one) and compare cost, latency and error rate in the dashboard.


Dashboard

Overview

Overview: cost, calls and what routing saved

Models

Models: providers, endpoints and pricing

Routers

Routers: tokens, budgets and routing per tenant

Usage

Usage: filter by time range, model and router

Playground

Playground: test in the browser, with routing trace and cost breakdown


Works with any OpenAI or Anthropic client

Routerly serves the OpenAI (/v1/chat/completions, /v1/responses) and Anthropic (/v1/messages) formats, so tools that speak either protocol work by changing the base URL.

ToolProtocolNotes
OpenAIPython, Node.js, .NET, Java, Go
AnthropicPython and Node.js SDKs, including streaming
OpenAISet the API base URL in Settings → Connections
Anthropic / OpenAICustom endpoint pointing to Routerly
OpenAIUse ChatOpenAI with base_url override
OpenAIOpenAI(api_base=...) constructor
OpenAIAdd a custom model via Settings → Models
OpenAIconfig.json, set apiBase to the Routerly URL
OpenAIConfigure as a custom OpenAI endpoint
Any fetch/curlOpenAI / AnthropicStandard HTTP, no SDK needed

Python (OpenAI SDK):

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:3000/v1",
    api_key="sk-rt-YOUR_ROUTER_TOKEN"   # printed by `routerly router create`
)
response = client.chat.completions.create(
    model="gpt-5-mini",
    messages=[{"role": "user", "content": "Hello!"}]
)

Python (Anthropic SDK):

import anthropic

client = anthropic.Anthropic(
    base_url="http://localhost:3000",
    api_key="sk-rt-YOUR_ROUTER_TOKEN"
)
message = client.messages.create(
    model="claude-haiku-4-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}]
)

Supported Providers

OpenAI, Anthropic, Google Gemini, Mistral, Cohere, xAI (Grok), DeepSeek, Groq, Together, Perplexity, Fireworks, Cerebras, Azure OpenAI, Amazon Bedrock, Google Vertex AI, Ollama (local), and any OpenAI-compatible endpoint as a custom provider. Subscription logins are available for OpenAI and Anthropic (openai-oauth, anthropic-oauth; see the guides); the unofficial session-cookie adapters (openai-web, anthropic-web) are configured as connections, see ChatGPT web and Claude web. The model catalog is fetched at runtime, so models and prices update without a new release.

A single router can span cloud and local models. Details: Providers.


Benchmarks

Independent benchmarks are published in the routerly-benchmark repository.


Documentation

Full documentation is available at https://doc.routerly.ai.

InstallationOne-line install, update channels, Docker, build from source
Quick StartFirst model, router and API call in 5 minutes
ConfigurationSettings file, environment variables
ArchitectureRouters, orchestrators, passthrough, request lifecycle, tracing
RoutersRouter configuration, tokens, members
RoutingAll 11 routing policies, profiles, selectors
Budgets & LimitsCost caps and enforcement
Security policiesGuardrails and PII scrubbing
OptimizersToken optimizers
ExperimentsA/B tests between routers
MCPMCP server and tokens
NotificationsChannels and events
DashboardWeb UI walkthrough
CLI ReferenceEvery command with examples
API ReferenceLLM proxy and management API
Self-Hosting GuideDocker, systemd, nginx, production checklist
IntegrationsCursor, Open WebUI, LangChain, LibreChat, and more

Configuration

All configuration lives in ~/.routerly/ (plain JSON files, no database). See Config files for the exact layout and Environment variables for overrides.

Override the base path with ROUTERLY_HOME=/custom/path. When running via Docker, configuration and data live in the routerly_data volume mounted at /data.


Security

Routerly is a TypeScript / Node.js project with no Python dependency. API keys are stored encrypted, router and MCP tokens only as SHA-256 hashes, passwords with bcrypt. See SECURITY.md for vulnerability reporting and supply chain notes.


Contributing

Contributions are welcome. See Releasing for version bumps and the maintainer release procedure.


Roadmap

Enterprise SSO

Log in to the dashboard with an existing identity provider (OAuth 2.0 / OIDC, LDAP). Not available yet: dashboard users are local accounts today.


License

Routerly is licensed under AGPL-3.0. For commercial licensing inquiries: [email protected]