๐ค AI Summary
Current agent leaderboards erroneously conflate task-specific performance with general capability, yielding unreliable deployment decisions. This work introduces four-facet generalizability theory to agent evaluation for the first time, employing variance decomposition to demonstrate that leaderboard rankings predominantly reflect task-specific expertise rather than general competence: main-effect variance is negligible, while interaction effects dominate. We estimate variance components using Henderson Method-I, REML (via lme4), and Bayesian binomial GLMMs, and analyze failure modes through MAST taxonomyโbased classification. Experiments across three major benchmarks reveal that evaluation reliability collapses on difficult tasks and that training-unit reliability negatively correlates with holdout reliability. To address these issues, we propose a reliability assessment framework tailored for enterprise deployment and introduce the DDR reporting standard. All code and data are publicly released to enable cross-benchmark diagnostic transfer.
๐ Abstract
Enterprise practitioners read agent leaderboards as if they ranked agent capability. We show, across three open agent-trace benchmarks (TheAgentCompany, $ฯ^2$-bench, and AppWorld), that the agent main effect accounts for less than 3% of total variance in every dataset and check type, while the agent-by-task interaction accounts for 7-23%. Leaderboards rank specialization, not capability. We arrive at this through a four-facet Generalizability Theory variance decomposition, fit with three estimators (Henderson Method-I, REML via lme4, and a Bayesian binomial GLMM) that agree to three decimal places. Four further findings sharpen what the leaderboard is hiding. First, aggregate reliability collapses on the hardest task quartile: $Eฯ^2$ on $ฯ^2$ action_checks falls from 0.752 to 0.000. Second, training-cell reliability negatively correlates with held-out reliability ($r = -0.90$ on $ฯ^2$), meaning the designs that look most reliable replicate worst. Third, population-level diagnostics transfer across enterprise benchmarks (capability-gap ratio stable at 0.35-0.40) but per-family agent rankings invert. Fourth, on the MAST failure taxonomy, trace-level mode profiles are idiosyncratic (MAE = 0.261) while cell-level profiles generalise (MAE = 0.056, $r = 0.83$). We package these into Deployment Decision Reliability (DDR), a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend. All code, data loaders, and fit artifacts are released under an open-source license.