Is Peer Review Really in Decline? Analyzing Review Quality across Venues and Time

2026-01-21

AI Summary

This work addresses widespread concerns in the academic community regarding declining peer review quality by proposing the first evaluation framework capable of enabling cross-conference and longitudinal comparisons of review quality. The framework introduces a method to characterize the diversity of review formats, employs standardized preprocessing pipelines, and integrates large language model–assisted scoring with lightweight heuristic metrics to construct a multidimensional quantitative assessment system. Empirical analysis applied to major AI/ML conferences—including ICLR, NeurIPS, and *ACL—reveals that median review quality does not exhibit a consistent downward trend, thereby challenging the prevailing narrative of “review quality decay.” The study further establishes a reproducible methodological foundation for future research on peer review mechanisms.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Peer review is at the heart of modern science. As submission numbers rise and research communities grow, the decline in review quality is a popular narrative and a common concern. Yet, is it true? Review quality is difficult to measure, and the ongoing evolution of reviewing practices makes it hard to compare reviews across venues and time. To address this, we introduce a new framework for evidence-based comparative study of review quality and apply it to major AI and machine learning conferences: ICLR, NeurIPS and *ACL. We document the diversity of review formats and introduce a new approach to review standardization. We propose a multi-dimensional schema for quantifying review quality as utility to editors and authors, coupled with both LLM-based and lightweight measurements. We study the relationships between measurements of review quality, and its evolution over time. Contradicting the popular narrative, our cross-temporal analysis reveals no consistent decline in median review quality across venues and years. We propose alternative explanations, and outline recommendations to facilitate future empirical studies of review quality.

Problem

• peer review
• review quality
• scientific publishing
• AI conferences
• temporal analysis

Innovation

• review quality
• peer review
• standardization
• large language models
• empirical analysis