Bayesian Inference and Decision Audits for Public Archives of Frontier AI Evaluations

2026-06-15

AI Summary

Public AI benchmarks are often misinterpreted as static leaderboards, overlooking their dynamic evolution shaped by reporting policies, benchmark revisions, and missing data—leading to erroneous assessments of the true state-of-the-art in model capabilities. This work reframes benchmark archives as a Bayesian inference problem and introduces the first framework integrating Bayesian reasoning with decision auditing. By reconstructing historical model trajectories, synthesizing posterior comparisons, modeling selection-aware frontiers, and calibrating uncertainty, the approach identifies unsupported claims of superiority and establishes empirically grounded temporal boundaries. Applied to archives such as LiveBench, the method recovers two distinct historical pathways, exposing systematic failures of current models in forecasting, preference transfer, and calibration, and effectively refutes overstated performance claims using a fixed audit threshold.

Abstract

Public AI evaluations are often read as terminal leaderboards, yet the underlying evidence is a selective time series shaped by reporting rules, benchmark revisions, and missingness. Repeated public archives for LiveBench and Open LLM Leaderboard v2 serve as the primary longitudinal record; LMArena provides a preference stress test; and GAIA and tau-bench contribute limited agentic pilots. Together, these archives instantiate a Bayesian inference problem: under a fixed reporting convention, one constructed terminal-only example over $1{,}000$ systems is compatible with two pre-terminal histories, yielding times of $23.03$ or $75.13$ to reach within $0.05$ of the ceiling under the same terminal-tail model. In synthetic posterior comparisons, action-facing diagnostics differ across observation regimes. The candidate selection-aware frontier model fails synthetic recovery, objective-archive prediction, preference transfer, and uncertainty calibration; correspondingly, fixed audit gates reject its stronger claims. An archive-and-adjudication protocol reconstructs public evaluation histories, isolates a verified timing boundary, and falsifies unsupported frontier claims.

Problem

• Bayesian inference
• public AI evaluations
• evaluation archives
• frontier claims
• missingness

Innovation

• Bayesian inference
• evaluation audit
• public AI benchmarks
• selection bias
• frontier model validation