Pitfalls of Evaluating Language Models with Open Benchmarks

2025-07-01

AI Summary

Open language model benchmarks (e.g., HELM, BIG-bench) suffer from a critical flaw—the public availability of test sets enables models to achieve spuriously high scores via test-set-only fine-tuning (“test-set overfitting”), undermining evaluation reliability and real-world validity. Method: The authors systematically construct compact models (BART, T5, GPT-2) and perform targeted fine-tuning exclusively on benchmark test sets; they rigorously assess generalization collapse via out-of-distribution evaluation. Contribution/Results: Such “cheating models” top public leaderboards yet fail catastrophically on unseen tasks. This work provides the first empirical demonstration of open benchmarks’ vulnerability to data leakage, exposing a fundamental threat to fair and meaningful model assessment. It advocates for private or dynamically updated evaluation protocols and calls for a paradigm shift toward robust, equitable, and practically relevant evaluation frameworks for language models.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Open Large Language Model (LLM) benchmarks, such as HELM and BIG-bench, offer standardized, transparent protocols that facilitate the fair comparison, reproducibility, and iterative advancement of Language Models (LMs). However, their openness also introduces critical and underexplored pitfalls. This study exposes these weaknesses by systematically constructing ``cheating'' models -- smaller variants of BART, T5, and GPT-2 fine-tuned directly on public test sets -- which achieve top rankings on a prominent open, holistic benchmark (HELM) despite poor generalization and limited practical utility. Our findings underscore three key insights: ca high leaderboard performance on open benchmarks may not always reflect real-world effectiveness; cb private or dynamic benchmarks must complement open evaluations to safeguard integrity; and cc a fundamental reevaluation of current benchmarking practices is essential to ensure robust and trustworthy LM assessments.

Problem

• Open LLM benchmarks may not reflect real-world model effectiveness
• Private or dynamic benchmarks are needed to ensure evaluation integrity
• Current benchmarking practices require reevaluation for trustworthy LM assessments

Innovation

• Fine-tuning small models on public test sets
• Using private or dynamic benchmarks for integrity
• Reevaluating benchmarking practices for robust assessments