What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review

2025-09-29

AI Summary

This study investigates key determinants of paper acceptance in open peer review, moving beyond static manuscript features to model the dynamic review process. Leveraging complete interaction data from over 28,000 submissions to ICLR 2017–2025, we integrate textual features, temporal reviewer–author behaviors, inter-reviewer disagreement metrics, and meta-review trajectories. Our analysis reveals—first time systematically—that response timeliness and interaction quality during the rebuttal phase exert a stronger influence on final decisions than initial review scores. Key findings indicate that clear writing, balanced figure usage, early submission, constructive and proactive author responses, and effective reconciliation of reviewer disagreements significantly increase acceptance probability. These results provide empirically validated, data-driven insights to enhance transparency, fairness, and efficiency in open peer review systems.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Learning Preferences or Rankings
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Security and Privacy: Data transparency and provenance

Abstract

Peer review is the primary mechanism for evaluating scientific contributions, yet prior studies have mostly examined paper features or external metadata in isolation. The emergence of open platforms such as OpenReview has transformed peer review into a transparent and interactive process, recording not only scores and comments but also rebuttals, reviewer-author exchanges, reviewer disagreements, and meta-reviewer decisions. This provides unprecedented process-level data for understanding how modern peer review operates. In this paper, we present a large-scale empirical study of ICLR 2017-2025, encompassing over 28,000 submissions. Our analysis integrates four complementary dimensions, including the structure and language quality of papers (e.g., section patterns, figure/table ratios, clarity), submission strategies and external metadata (e.g., timing, arXiv posting, author count), the dynamics of author-reviewer interactions (e.g., rebuttal frequency, responsiveness), and the patterns of reviewer disagreement and meta-review mediation (e.g., score variance, confidence weighting). Our results show that factors beyond scientific novelty significantly shape acceptance outcomes. In particular, the rebuttal stage emerges as a decisive phase: timely, substantive, and interactive author-reviewer communication strongly increases the likelihood of acceptance, often outweighing initial reviewer skepticism. Alongside this, clearer writing, balanced visual presentation, earlier submission, and effective resolution of reviewer disagreement also correlate with higher acceptance probabilities. Based on these findings, we propose data-driven guidelines for authors, reviewers, and meta-reviewers to enhance transparency and fairness in peer review. Our study demonstrates that process-centric signals are essential for understanding and improving modern peer review.

Problem

• Analyzes factors driving paper acceptance in modern peer review systems
• Examines author-reviewer interaction dynamics and rebuttal impact on decisions
• Investigates how review process signals complement scientific novelty in outcomes

Innovation

• Analyzed peer review using multi-dimensional process data
• Integrated submission strategies and interaction dynamics
• Proposed data-driven guidelines for review fairness