A Survey on Large Language Model Benchmarks

2025-08-21

AI Summary

Current LLM benchmarks suffer from pervasive data contamination, cultural-linguistic bias, lack of procedural transparency, and insufficient dynamism, leading to unreliable evaluations. To address this, we conduct the first systematic survey of 283 mainstream LLM benchmarks, proposing a three-dimensional taxonomy—spanning general capabilities, domain-specific competencies, and goal-specific functionalities—that encompasses language understanding, knowledge reasoning, natural sciences, social sciences and humanities, and risk controllability. Through empirical analysis, we uncover structural biases in evaluation objectives, data provenance, and assessment methodologies. Our key contributions are: (1) the first comprehensive benchmark taxonomy map; (2) identification of critical assessment deficiencies; and (3) a novel benchmark design paradigm grounded in trustworthiness, fairness, and adaptability. This work establishes both a theoretical framework and practical guidelines for developing high-fidelity, next-generation LLM evaluation systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

In recent years, with the rapid development of the depth and breadth of large language models' capabilities, various corresponding evaluation benchmarks have been emerging in increasing numbers. As a quantitative assessment tool for model performance, benchmarks are not only a core means to measure model capabilities but also a key element in guiding the direction of model development and promoting technological innovation. We systematically review the current status and development of large language model benchmarks for the first time, categorizing 283 representative benchmarks into three categories: general capabilities, domain-specific, and target-specific. General capability benchmarks cover aspects such as core linguistics, knowledge, and reasoning; domain-specific benchmarks focus on fields like natural sciences, humanities and social sciences, and engineering technology; target-specific benchmarks pay attention to risks, reliability, agents, etc. We point out that current benchmarks have problems such as inflated scores caused by data contamination, unfair evaluation due to cultural and linguistic biases, and lack of evaluation on process credibility and dynamic environments, and provide a referable design paradigm for future benchmark innovation.

Problem

• Evaluating inflated scores from data contamination in benchmarks
• Addressing unfair cultural and linguistic biases in evaluation
• Assessing lack of process credibility and dynamic environment tests

Innovation

• Systematic categorization of 283 benchmarks
• Identified data contamination and bias issues
• Proposed design paradigm for future benchmarks