Cost-Optimal LLM Routing with Limited User Feedback under User Satisfaction Guarantees

2026-06-12

AI Summary

This work addresses the tension between escalating inference costs and stringent service-level agreement (SLA) requirements in large language model (LLM) deployment, where existing routing approaches struggle to simultaneously achieve cost efficiency and SLA compliance under sparse, one-sided user feedback. To this end, we propose SLARouter—the first online LLM routing mechanism that operates without offline training or task-specific tuning. Built upon a constrained multi-armed bandit framework, SLARouter dynamically optimizes routing decisions using only sparse satisfaction signals observed in production environments. Theoretical analysis guarantees both asymptotic cost optimality and strict SLA adherence. Empirical evaluations across multiple benchmarks demonstrate that SLARouter consistently meets SLA targets without hyperparameter tuning and reduces inference costs by up to 2.2× compared to state-of-the-art methods.

Abstract

Inference costs for large language model (LLM) applications are rapidly growing, driven by surging demand and rising infrastructure cost. Users expect high-quality responses, and in commercial settings this is formally codified in Service Level Agreements (SLAs), creating a fundamental tension between cost and quality. Recent progress on cost-aware LLM request routing has shown potential to resolve this tension, but existing approaches rely on complete feedback signals, offline training, extensive per-workload tuning, and most lack SLA guarantees or inference-time adaptivity. We introduce SLARouter, an online routing algorithm that learns a cost-optimal policy from the sparse, one-sided user feedback available in production systems. SLARouter provides theoretical guarantees for both cost optimality and strict SLA compliance. Experiments across a wide range of LLM benchmarks show that SLARouter satisfies SLA constraints without the need for per-benchmark tuning, reducing operating cost by up to 2.2x over existing baselines.

Problem

• LLM routing
• cost optimization
• user feedback
• SLA guarantees
• inference cost

Innovation

• LLM routing
• cost optimization
• SLA guarantees
• online learning
• sparse feedback