Recommendations and Reporting Checklist for Rigorous&Transparent Human Baselines in Model Evaluations

2025-06-09

AI Summary

Current human baselines in large language model evaluation lack methodological rigor and transparency, undermining the validity of claims such as “superhuman performance.” Method: This paper pioneers the systematic integration of classical measurement theory into AI evaluation, establishing a comprehensive theoretical framework spanning human baseline design, execution, and reporting. It introduces an actionable quality assessment system and a standardized checklist, derived via meta-review–driven framework development, structured checklist design, and empirical systematic auditing. Contribution/Results: Applying this framework to diagnose 115 human baseline studies, we identify pervasive methodological flaws. The resulting open-source audit tool significantly enhances reproducibility, comparability, and accountability in AI evaluation. By grounding benchmarking practice in psychometric principles, our work provides a rigorous methodological foundation for scientifically credible model capability assessment.

Technology Category

• Humans and AI: Other Foundations of Human Computation & AI
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

In this position paper, we argue that human baselines in foundation model evaluations must be more rigorous and more transparent to enable meaningful comparisons of human vs. AI performance, and we provide recommendations and a reporting checklist towards this end. Human performance baselines are vital for the machine learning community, downstream users, and policymakers to interpret AI evaluations. Models are often claimed to achieve"super-human"performance, but existing baselining methods are neither sufficiently rigorous nor sufficiently well-documented to robustly measure and assess performance differences. Based on a meta-review of the measurement theory and AI evaluation literatures, we derive a framework with recommendations for designing, executing, and reporting human baselines. We synthesize our recommendations into a checklist that we use to systematically review 115 human baselines (studies) in foundation model evaluations and thus identify shortcomings in existing baselining methods; our checklist can also assist researchers in conducting human baselines and reporting results. We hope our work can advance more rigorous AI evaluation practices that can better serve both the research community and policymakers. Data is available at: https://github.com/kevinlwei/human-baselines

Problem

• Improve rigor in human baselines for AI evaluations
• Enhance transparency in reporting human vs AI performance
• Address shortcomings in existing baselining methods

Innovation

• Framework for rigorous human baseline design
• Checklist for transparent baseline reporting
• Meta-review of measurement theory literature