Rebuttals Move Peer-Review Scores, but Initial-Review Structure Bounds the Movement

2026-06-20

AI Summary

This study addresses the challenge of quantifying the actual impact of author rebuttals on peer review scores, which is confounded by initial ratings, reviewer consensus, confidence levels, and dynamic discussion effects. Leveraging 73,000 review trajectories from ICLR 2024–2025, the authors employ large language models—including Gemini Flash 3.0 and Claude Opus 4.6—to predict implicit initial scores from score-stripped review texts and construct a 44-dimensional feature space capturing rebuttal interactions. Using externally archived pre- and post-rebuttal scores and Bonferroni-corrected cross-year validation, they isolate the effect of rebuttal content for the first time. Results show that the initial review structure strongly constrains score changes (AUC = 0.747), with performance improving to 0.804 upon incorporating rebuttal features. Notably, when rebuttal quality exceeds the original score’s expectation, upward revisions occur in 31.9% of cases—significantly higher than the 8.3% rate for low-quality rebuttals.

Abstract

Author rebuttals are the main post-submission window in peer review, but their effect on reviewer scores remains hard to measure because score updates mix rebuttal content with initial score position, paper-level consensus, reviewer confidence, and discussion dynamics. We study ICLR 2024-2025 using 73,000 reviewer trajectories with externally archived pre- and post-rebuttal scores, and use LLMs only as measurement instruments. Gemini Flash 3.0 predicts implied pre-rebuttal scores from score-stripped review text. The resulting text-score offset predicts later movement, with score-increase rates rising from 8.3% when text reads below the assigned score to 31.9% when it reads above. Claude Opus 4.6 induces, and outcome-blinded Gemini Flash 3.0 validates, a 44-feature taxonomy of resolved reviewer-author exchanges, where 23 features replicate across model and held-out year under Bonferroni correction. In the rebuttal-engaged benchmark (n=6,705), initial-review structure already predicts much score movement (AUC=0.747, minimal AUC=0.696), while adding the resolved exchange raises AUC to 0.804. Rebuttals can move scores, but measurable movement is bounded by initial-review structure, and robust exchange signals are mostly rebuttal failure modes.

Problem

• peer review
• author rebuttals
• reviewer scores
• score movement
• initial-review structure

Innovation

• author rebuttal
• peer review dynamics
• LLM as measurement instrument
• score-text offset
• rebuttal failure modes