The AI Imperative: Scaling High-Quality Peer Review in Machine Learning

2025-06-09

AI Summary

The surge in submissions to top-tier machine learning conferences, coupled with a shortage of qualified reviewers, has degraded review quality, reduced inter-reviewer consistency, and exacerbated reviewer fatigue. To address this, we propose an AI-augmented peer review collaboration ecosystem, positioning large language models (LLMs) as collaborative partners—not replacements—for authors, reviewers, and area chairs (ACs). We introduce the first systematic AI-assisted review research framework, grounded in structured, fine-grained, and ethically compliant review data. It defines four core application scenarios: factual verification, reviewer guidance, author feedback optimization, and AC decision support. Our framework further encompasses reviewer behavior modeling, multi-role collaborative interfaces, and a principled data governance architecture. Preliminary experimental validation confirms feasibility. This work provides both theoretical foundations and practical blueprints for building scalable, trustworthy, and human-centered next-generation review infrastructure.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Peer review, the bedrock of scientific advancement in machine learning (ML), is strained by a crisis of scale. Exponential growth in manuscript submissions to premier ML venues such as NeurIPS, ICML, and ICLR is outpacing the finite capacity of qualified reviewers, leading to concerns about review quality, consistency, and reviewer fatigue. This position paper argues that AI-assisted peer review must become an urgent research and infrastructure priority. We advocate for a comprehensive AI-augmented ecosystem, leveraging Large Language Models (LLMs) not as replacements for human judgment, but as sophisticated collaborators for authors, reviewers, and Area Chairs (ACs). We propose specific roles for AI in enhancing factual verification, guiding reviewer performance, assisting authors in quality improvement, and supporting ACs in decision-making. Crucially, we contend that the development of such systems hinges on access to more granular, structured, and ethically-sourced peer review process data. We outline a research agenda, including illustrative experiments, to develop and validate these AI assistants, and discuss significant technical and ethical challenges. We call upon the ML community to proactively build this AI-assisted future, ensuring the continued integrity and scalability of scientific validation, while maintaining high standards of peer review.

Problem

• AI-assisted peer review to handle increasing ML paper submissions
• Improving review quality and consistency with AI-human collaboration
• Developing ethical AI tools for peer review data and processes

Innovation

• AI-augmented ecosystem with LLMs
• AI enhances factual verification and guidance
• Structured ethically-sourced peer review data