The Capability Frontier: Benchmarks Miss 82% of Model Performance

2026-06-25

AI Summary

Current large language model benchmarks systematically underestimate true model capabilities under heterogeneous data distributions by evaluating only a single model in a single execution. This work proposes the “capability frontier” evaluation paradigm, which characterizes the optimal performance achievable through multi-model routing and multi-generation selection across varying computational budgets using Pareto frontiers. Leveraging oracle-based optimal selection, controlled probabilistic simulation, and a cross-task benchmark spanning 16 domains—including programming, reasoning, and medicine—the study quantifies, for the first time, the performance underestimation inherent in conventional evaluation protocols. Experiments demonstrate that the proposed approach reduces error rates by 54% and improves overall performance by 82% compared to single-model single-run baselines, while achieving state-of-the-art accuracy at 85% lower computational cost.

Abstract

Existing benchmarks typically report accuracy for a single model on a single run. This systematically understates real-world LLM capabilities, particularly under heterogeneous data distributions: (i) different models get different questions correct according to their specializations, and (ii) given a budget, multiple generations can be sampled and selectively retained. To quantify this gap, we introduce the Capability Frontier: a Pareto frontier over a set of models that characterizes the best achievable performance at each cost level under optimal selection across models and generations (i.e., via an oracle). Our construction corrects for two opposing biases: underestimation from single-model evaluation and overestimation from taking maxima over noisy samples. We study 21 LLMs across 16 widely used benchmarks spanning coding, reasoning, medicine, factuality, instruction following, and agentic tasks, comparing Capability Frontier performance at matched cost to each benchmark's top-performing model. Correcting for single-model evaluation yields a 54% error rate reduction; additionally correcting for single runs yields an 82% improvement, with SOTA accuracy matched at 85% cost reduction. Complementing these empirical results, we use controlled probabilistic simulations to show that higher query topic entropy produces a near-monotonic increase in the performance gap between oracle routing and the best single model. Our findings suggest collective LLM capabilities are substantially underestimated, with implications for evaluation and deployment in data-heterogeneous, multi-domain settings.

Problem

• benchmarking
• large language models
• capability evaluation
• heterogeneous data
• model performance

Innovation

• Capability Frontier
• Pareto frontier
• oracle routing
• heterogeneous data
• LLM evaluation