🤖 AI Summary
This study addresses the substantial overestimation of the “unsolvability ceiling” in multi-LLM routing observed in prior work, which stems from evaluation artifacts such as judge bias, generation truncation, and format mismatches, thereby misleading router design. The authors propose the first systematic decomposition framework that leverages dual-judge verification, exact-match anchoring, and randomized shuffling controls. Applying this framework to 206,000 query–model pairs across six benchmarks using Gemma 4 and Llama 3.1 series models, they rigorously quantify how these artifacts inflate perceived unsolvability and distort training signals for routers. Their analysis markedly reduces the measured unsolvable fractions across tasks and reveals that standard routers often degenerate into majority-class predictors, incurring opportunity cost losses of 13–17 percentage points.
📝 Abstract
Efficient routing across multiple LLMs enables cost-quality tradeoffs by directing queries to the cheapest capable model. Prior work attributes routing headroom to an "unsolvability ceiling", queries no model in the pool can solve. We present a large-scale study of multi-tier LLM routing with 206,000 query-model pairs across six benchmarks (MMLU, MedQA, HumanEval, MBPP, Alpaca, ShareGPT) using the Gemma 4 and Llama 3.1 families. Evaluating with both LLM-as-a-judge and exact-match metrics, we show that a substantial portion of reported unsolvability stems from evaluation artifacts: (i) systematic judge biases favoring verbosity over correctness, (ii) truncation under fixed generation budgets, and (iii) output format mismatches. Through dual-judge validation and exact-match grounding, we reduce measured unsolvability across tasks. We introduce a decomposition framework attributing failures to these artifacts, revealing consistent patterns across domains and model families. These artifacts also distort router training signals: standard routers collapse to majority-class prediction (~79% smallest-tier optimal), confirmed via random-feature and shuffled-label controls, incurring a 13-17 percentage point opportunity cost. We provide actionable recommendations including dual-judge validation, exact-match anchoring, and cost-sensitive objectives. Our findings suggest existing routing headroom estimates are substantially inflated, underscoring the need for reliable evaluation protocols in multi-LLM systems.