Insights from the ICLR Peer Review and Rebuttal Process

2025-11-19

AI Summary

To address the dual challenge of improving review efficiency and quality at top-tier conferences such as ICLR, this paper systematically investigates rebuttal-stage score dynamics and reviewer–author interaction mechanisms using large-scale peer-review data from 2024–2025. Methodologically, it pioneers an integrated approach combining statistical modeling (time-series analysis and regression forecasting) with large language model (LLM)-based text classification for fine-grained pattern mining in review–rebuttal discourse. Key contributions include: (1) identifying initial scores and co-reviewer feedback as the strongest predictors of score changes; (2) demonstrating that high-quality author rebuttals significantly increase acceptance probability for borderline papers; and (3) uncovering high-impact rebuttal strategies and latent inter-reviewer influence pathways. The framework enables interpretable, dynamic modeling of interactive features through LLM-augmented textual analysis. All code and datasets are publicly released.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Calibration & Uncertainty Quantification
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Peer review is a cornerstone of scientific publishing, including at premier machine learning conferences such as ICLR. As submission volumes increase, understanding the nature and dynamics of the review process is crucial for improving its efficiency, effectiveness, and the quality of published papers. We present a large-scale analysis of the ICLR 2024 and 2025 peer review processes, focusing on before- and after-rebuttal scores and reviewer-author interactions. We examine review scores, author-reviewer engagement, temporal patterns in review submissions, and co-reviewer influence effects. Combining quantitative analyses with LLM-based categorization of review texts and rebuttal discussions, we identify common strengths and weaknesses for each rating group, as well as trends in rebuttal strategies that are most strongly associated with score changes. Our findings show that initial scores and the ratings of co-reviewers are the strongest predictors of score changes during the rebuttal, pointing to a degree of reviewer influence. Rebuttals play a valuable role in improving outcomes for borderline papers, where thoughtful author responses can meaningfully shift reviewer perspectives. More broadly, our study offers evidence-based insights to improve the peer review process, guiding authors on effective rebuttal strategies and helping the community design fairer and more efficient review processes. Our code and score changes data are available at https://github.com/papercopilot/iclr-insights.

Problem

• Analyzing ICLR peer review dynamics to improve efficiency and paper quality
• Identifying factors influencing score changes during author rebuttal process
• Understanding reviewer interactions and their impact on final acceptance decisions

Innovation

• Large-scale analysis of peer review dynamics
• LLM-based categorization of review texts
• Quantitative identification of effective rebuttal strategies