Benchmarking is Broken - Don't Let AI be its Own Judge

2025-10-08

AI Summary

Contemporary AI benchmarking suffers from pervasive data contamination, selective reporting, and inadequate quality control—leading to severe evaluation bias, diminished scientific signal, and eroded public trust. To address these challenges, this paper introduces PeerBench: a novel evaluation paradigm featuring a sealed execution environment, dynamically rotated item banking, delayed result disclosure, and decentralized governance. Its core innovation lies in institutionalizing AI evaluation as a dynamic, auditable, and community-governed process—shifting from ad hoc practice to a robust, evolving trust infrastructure. Empirical evaluation demonstrates that PeerBench significantly enhances assessment fairness and reproducibility, effectively distinguishing genuine progress from inflated claims. By establishing standardized, transparent, and sustainable evaluation infrastructure, PeerBench advances the rigor, accountability, and long-term viability of AI benchmarking.

Technology Category

• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Humans and AI: Learning Human Values and Preferences

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The meteoric rise of Artificial Intelligence (AI), with its rapidly expanding market capitalization, presents both transformative opportunities and critical challenges. Chief among these is the urgent need for a new, unified paradigm for trustworthy evaluation, as current benchmarks increasingly reveal critical vulnerabilities. Issues like data contamination and selective reporting by model developers fuel hype, while inadequate data quality control can lead to biased evaluations that, even if unintentionally, may favor specific approaches. As a flood of participants enters the AI space, this "Wild West" of assessment makes distinguishing genuine progress from exaggerated claims exceptionally difficult. Such ambiguity blurs scientific signals and erodes public confidence, much as unchecked claims would destabilize financial markets reliant on credible oversight from agencies like Moody's. In high-stakes human examinations (e.g., SAT, GRE), substantial effort is devoted to ensuring fairness and credibility; why settle for less in evaluating AI, especially given its profound societal impact? This position paper argues that the current laissez-faire approach is unsustainable. We contend that true, sustainable AI advancement demands a paradigm shift: a unified, live, and quality-controlled benchmarking framework robust by construction, not by mere courtesy and goodwill. To this end, we dissect the systemic flaws undermining today's AI evaluation, distill the essential requirements for a new generation of assessments, and introduce PeerBench, a community-governed, proctored evaluation blueprint that embodies this paradigm through sealed execution, item banking with rolling renewal, and delayed transparency. Our goal is to pave the way for evaluations that can restore integrity and deliver genuinely trustworthy measures of AI progress.

Problem

• Current AI benchmarks reveal critical vulnerabilities and lack unified evaluation standards
• Data contamination and selective reporting create biased assessments and exaggerated claims
• Inadequate evaluation methods blur scientific progress and erode public trust in AI

Innovation

• Community-governed proctored evaluation blueprint
• Sealed execution with delayed transparency
• Item banking with rolling renewal