Beyond Static Leaderboards: Predictive Validity for the Evaluation of LLM Agents

2026-06-17

AI Summary

Current large-model agent benchmarks rely on static average-score leaderboards, which poorly predict real-world performance in out-of-distribution deployment scenarios. This work proposes a deployment-oriented, multidimensional evaluation framework centered on predictive validity—the correlation between in-sample and out-of-distribution rankings. We introduce a novel evaluation paradigm grounded in predictive validity, featuring a twelve-tier measurement architecture and three falsifiable criteria for out-of-distribution assessment. Through a preregistered empirical study integrating 14 parallel implementations, seven existing benchmarks, and extensions across multimodal settings, diverse agent orchestrations, and retrieval-augmented approaches, we demonstrate that conventional leaderboards yield unstable rankings under distribution shift, whereas our paradigm exhibits significantly stronger predictive power, thereby establishing a methodological foundation for next-generation agent benchmarks.

Abstract

Agent benchmarks are growing fast, but no single benchmark touches more than four or five of the dimensions that deployment exposes. This paper aggregates the largest coordinated deep-dive of one MCP-based industrial-agent benchmark to date: fourteen parallel implementation studies covering new asset classes (including a multi-modal visual extension), alternative orchestrations, retrieval strategies, reasoning modes, infrastructure optimizations, and evaluation-methodology probes. Consolidating those studies with seven prior agent benchmarks, we argue that aggregate-score leaderboards systematically underspecify deployed-agent evaluation. Rankings derived from aggregate scores do not transfer to out-of-distribution settings; recent public-to-hidden competition retrospectives provide direct empirical evidence of this rank instability. We propose ranking configurations by predictive validity, the correlation between in-sample and out-of-sample rank, rather than in-sample mean, and report a twelve-tier measurement apparatus that exposes the deployment-relevant dimensions HELM and its agent-era successors collapse. The position is operationalized through three falsifiable out-of-distribution criteria with explicit thresholds; existing evidence partly supports it but is too thin to confirm. We close with a pre-registered pilot design and a field-level vision for what the next generation of agentic benchmarks should report.

Problem

• predictive validity
• LLM agents
• benchmarking
• out-of-distribution
• leaderboards

Innovation

• predictive validity
• out-of-distribution evaluation
• agent benchmarking
• LLM agents
• measurement framework