CARGO: A Framework for Confidence-Aware Routing of Large Language Models

2025-09-18

AI Summary

In multi-LLM deployment, prompt routing must jointly optimize performance, cost, and latency. Method: We propose a lightweight, confidence-aware two-stage dynamic routing framework. Stage I employs an embedding-based regressor to estimate relative model performance per task; Stage II deploys an uncertainty-triggered binary classifier for fine-grained category assignment (math, programming, reasoning, summarization, creative writing), trained exclusively on LLM-generated pairwise comparison signals—no human annotations required. Contribution/Results: Our key innovation is a category-aware routing mechanism optimized for performance gaps, enabling domain-specific modeling. Experiments across four state-of-the-art LLMs achieve 76.4% Top-1 routing accuracy and increase win rates over single-expert baselines to 72%–89%, significantly enhancing deployment efficiency and practicality of multi-model systems.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

As large language models (LLMs) proliferate in scale, specialization, and latency profiles, the challenge of routing user prompts to the most appropriate model has become increasingly critical for balancing performance and cost. We introduce CARGO (Category-Aware Routing with Gap-based Optimization), a lightweight, confidence-aware framework for dynamic LLM selection. CARGO employs a single embedding-based regressor trained on LLM-judged pairwise comparisons to predict model performance, with an optional binary classifier invoked when predictions are uncertain. This two-stage design enables precise, cost-aware routing without the need for human-annotated supervision. To capture domain-specific behavior, CARGO also supports category-specific regressors trained across five task groups: mathematics, coding, reasoning, summarization, and creative writing. Evaluated on four competitive LLMs (GPT-4o, Claude 3.5 Sonnet, DeepSeek V3, and Perplexity Sonar), CARGO achieves a top-1 routing accuracy of 76.4% and win rates ranging from 72% to 89% against individual experts. These results demonstrate that confidence-guided, lightweight routing can achieve expert-level performance with minimal overhead, offering a practical solution for real-world, multi-model LLM deployments.

Problem

• Routing user prompts to optimal LLMs for performance and cost
• Predicting model performance without human-annotated supervision
• Achieving expert-level routing accuracy across diverse task domains

Innovation

• Lightweight confidence-aware framework for LLM routing
• Embedding-based regressor predicts model performance
• Optional binary classifier handles uncertain predictions