How Can Rhetoric Reward-Hack AI Reviewers? Dissecting Rhetorical Sensitivity in AI-Based Peer Review

2026-08-09

AI Summary

This study investigates how rhetorical strategies influence the scoring behavior of large language model (LLM)-driven AI peer reviewers when scientific content remains unchanged, thereby exposing potential "reward hacking" risks. By constructing a controlled corpus of 4,200 papers and systematically manipulating six rhetorical dimensions—augmented with adversarial rewriting techniques (joint, recursive, and reviewer-guided) and multi-protocol review mechanisms—the work quantifies, for the first time, the sensitivity hierarchy of AI reviewers to rhetoric, identifying evidence framing and novelty positioning as the most impactful factors. Results demonstrate that rhetorical adjustments can significantly alter AI-assigned scores, with low-scoring papers easily elevated and high-scoring ones readily downgraded. While rewriters primarily drive variant differences, reviewers determine the direction and magnitude of score shifts; stringent reviewing lowers average scores but does not alter the underlying sensitivity pattern.

Abstract

As large language models increasingly participate in scientific evaluation, we investigate a potential form of reward hacking: how rhetorical choices shape AI-review judgments when reported scientific content is preserved and how these effects vary across evaluation conditions. We construct a controlled corpus of 4,200 full-paper manuscripts derived from 120 anonymized ICLR 2026 submissions. Two LLM rewriters transform six rhetorical dimensions in opposing directions, and five LLM reviewers evaluate the resulting manuscripts under standard and strict protocols. We also test joint, recursive, and reviewer-guided rewriting. Our results show that rhetorical sensitivity is structured rather than uniform. Evidence framing and novelty stance produce the largest positive-negative contrasts in overall assessment, with scope framing forming a weaker second tier; the remaining dimensions have smaller or less stable effects. This hierarchy persists across human-assessed quality levels, but score movement depends strongly on the AI reviewer's original score: lower scores tend to rise, higher scores tend to fall, and directional contrasts are clearest in the middle ranges. More elaborate workflows do not reliably yield larger gains. Joint rewriting is strongly rewriter-dependent, reviewer guidance does not consistently outperform an unguided second pass, and repeated rewriting yields diminishing, configuration-dependent returns. Across conditions, the rewriter primarily determines the separation between opposing variants, whereas the reviewer determines the magnitude and sign of their score effects. Strict review lowers mean OA by 1.36 points without consistently changing rhetorical sensitivity. These findings identify when rhetorical presentation influences AI scientific review and motivate evaluation systems robust to content-preserving variation in scientific writing.

Problem

• rhetorical sensitivity
• AI peer review
• reward hacking
• scientific evaluation
• large language models

Innovation

• rhetorical sensitivity
• AI peer review
• reward hacking
• controlled rewriting
• LLM evaluation