Cluster, Route, Escalate: Cascaded Framework for Cost-Aware LLM Serving

2026-06-25

AI Summary

This work addresses the challenge of balancing inference cost and accuracy in large language model deployment, where a single model struggles to handle queries of varying difficulty efficiently. The authors propose a two-stage cascaded framework: first, a cost-aware routing mechanism assigns incoming queries to the most cost-effective model based on query clustering; second, an unsupervised quality estimation algorithm automatically triggers reprocessing by a stronger model when the initial output is deemed low-quality. The system employs offline, interpretable hyperparameter tuning to control computational budget and enables adaptive model switching without human intervention. Experimental results demonstrate that the approach maintains 97–99% of the top-tier model’s accuracy across multiple benchmarks while significantly reducing time-per-output-token (TPOT).

Abstract

Efficient deployment of large language models (LLMs) in production forces a trade-off between accuracy and cost. Operators often default to a single model that is either expensive for easy queries or insufficient for hard ones. To address this challenge, we propose a two-stage cascaded solution. Stage 1 clusters incoming queries and assigns each cluster to its most cost-effective model. The cost budget for this routing process is set by an interpretable hyperparameter, tuned offline. Stage 2 adds a quality estimation (QE) cascade; when an output from Stage 1 is judged low-quality, the query is escalated to a stronger model. This ensures only hard or low-confidence cases reach the expensive models. On the test datasets, the cascaded system retains 97-99% of the strongest model's accuracy while reducing Time Per Output Token (TPOT). It requires only task-correctness labels and adapts to changes in the model pool without manual reconfiguration.

Problem

• cost-aware LLM serving
• accuracy-cost trade-off
• query routing
• model cascading
• efficient deployment

Innovation

• cost-aware LLM serving
• query clustering
• quality estimation
• model cascading
• adaptive routing