olla
High-performance lightweight proxy and load balancer for LLM infrastructure. Intelligent routing, automatic failover and unified model discovery across local and remote inference backends.
Automatically checked · GitHub checked 2026-11-17

Overview
Olla is a high-performance, low-overhead, low-latency proxy and load balancer for managing LLM infrastructure. It intelligently routes LLM requests across self-hosted inference nodes with a wide variety of natively supported endpoints and extensible enough to support others. Olla provides model discovery and unified model catalogues within each provider, enabling seamless routing to available models on compatible endpoints. Olla works alongside API gateways like LiteLLM or orchestration platforms like GPUStack, focusing on making your existing LLM infrastructure reliable through intelligent routing and failover. You can choose between two proxy engines: Sherpa for simplicity and maintainability or Olla for maximum performance with advanced features like circuit breakers and connection pooling. Single CLI application and config file is all you need to go Olla! For large GPU deployments, enterprise and data centre use, see TensorFoundry FoundryOS. For an inference control plane, consider Alloy. - 🔄 Smart Load Balancing: Priority-based routing with automatic failover and connection retry - 📌 Sticky Sessions: KV-cache-aware affinity routing that pins multi-turn conversations to the same backend - 🔍 Smart Model Unification: Per-provider unification + OpenAI-compatible cross-provider routing - ⚡ Dual Proxy Engines: Sherpa (simple) and Olla (high-performance) - 🎯 Advanced Filtering: Profile and model filtering with glob patterns for precise control - 💊 Health Monitoring: Continuous endpoint health checks with circuit breakers and automatic recovery - 🔁 Intelligent Retry: Automatic retry on connection failures with immediate transparent endpoint failover, tunable with proxy.retry - 🔧 Self-Healing: Automatic model discovery refresh when endpoints recover - 📊 Request Tracking: Detailed response headers and statistics - ⚡🔄 Anthropic Messages API: Passthrough for backends with native support; automatic translation for others - 🧬 Embeddings and Rerank: Text Embeddings Inference support, with model-less embedding and rerank requests routed to a host serving that kind - 🔐 Backend Authentication: Outbound auth for local backends requiring API keys, bearer tokens, or basic auth - 🛡️ Production Ready: Rate limiting, request size limits, optional CORS for browser clients, configurable proxy/server timeouts, graceful shutdown - ⚡ High Performance: Sub-millisecond endpoint selection with lock-free atomic stats - 🎯 LLM-Optimised: Streaming-first design with optimised timeouts for long inference