Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation

2026-08-07

AI Summary

This study addresses the trust crisis in large language model (LLM) evaluation stemming from non-verifiable results, identity bias, and selective reporting. It presents the first systematic quantification of how model identity influences human ratings, employing a controlled experiment comparing blind versus identity-disclosed evaluations across seven open-source LLMs—including GPT-OSS 120B—on diverse tasks. The authors propose a decentralized evaluation framework built on an Ethereum-compatible blockchain, integrating autonomous economic agents with a commit-reveal protocol to ensure an auditable and tamper-resistant assessment pipeline. Empirical results demonstrate that disclosing model identity significantly alters scores on geosensitive tasks—for instance, GLM5.1 gains 7.00 points (p = 0.0249)—thereby validating the efficacy and novelty of the proposed mechanism.

Abstract

LLM benchmarks can build an organization's reputation and attract customers, but only when results are transparent and verifiable. Unverified claims that DeepSeek R1 outperformed OpenAI's o1 contributed to market panic on January 27, 2025, when Nvidia lost USD589 billion in market value. Yet vendor benchmarks often depend on an honor system. Academic reassessments and independent leaderboards have found undisclosed changes to proprietary models, contaminated training data, and selective reporting. LLM-as-a-judge methods scale evaluation by reducing human review. Studies, however, suggest that judges may show identity-aware bias, scoring an answer according to its source model rather than its quality. This bias has not been fully measured or corrected across politically sensitive, reasoning-intensive, and preference-based tasks. We examine this problem using seven verifier models: GPT-OSS 120B, Llama 3.3 70B, GLM 5.1, Qwen3 32B, DeepSeek V4 Pro, Mistral Large3, and Sarvam M. They score anonymous and identity-disclosed responses from three primary models on 58 factual, reasoning, political, and preference-based questions. Identity disclosure slightly raises scores for factual questions, moderately affects stress-reasoning tasks, and causes large changes for geopolitically sensitive topics. Notable results include GLM5.1 (+7.00 points, p = 0.0249) and Llama 3.3 70B (+1.56 points, p = 0.00). We also introduce a blockchain-based commit-reveal protocol using Autonomous Economic Agents on an Ethereum-compatible ledger. In Phase 1, each judge records a one-way hash of its score and a secret salt before candidate identities are revealed. In Phase 2, the identity and raw score are disclosed and verified on-chain. This creates a tamper-evident audit trail that separates blind evaluation from post-hoc claims and reduces the verification burden on independent researchers and leaderboard operators.

Problem

• LLM evaluation
• verifiability
• identity-aware bias
• benchmark transparency
• trust decentralization

Innovation

• identity-aware bias
• LLM-as-a-judge
• commit-reveal protocol
• blockchain-based evaluation
• tamper-evident audit