REMOR: Automated Peer Review Generation with LLM Reasoning and Multi-Objective Reinforcement Learning

2025-05-16

AI Summary

Existing AI peer-review systems often produce superficial, overly laudatory feedback lacking depth and critical analysis. Method: We propose REMOR, a framework leveraging multi-objective reinforcement learning and a reasoning-augmented large language model (DeepSeek-R1-Distill-Qwen-7B) to learn a human-aligned, multi-dimensional reward function—HPRR—and introduce PeerRT, the first high-quality peer-review dataset with explicit chain-of-thought rationales. Contribution/Results: Contrary to expectations, the uniformly weighted variant REMOR-U outperforms HPRR in substantive feedback quality, achieving significant gains in depth, criticality, and information density over baselines. Experiments show REMOR-U/H attains average rewards 2.1× those of human reviewers, with review quality comparable to top-tier human reviewers while effectively suppressing low-quality, long-tail outputs. All code, the HPRR reward function, the PeerRT dataset, and trained models are publicly released.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Knowledge Representation and Reasoning: Qualitative Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

AI-based peer review systems tend to produce shallow and overpraising suggestions compared to human feedback. Here, we evaluate how well a reasoning LLM trained with multi-objective reinforcement learning (REMOR) can overcome these limitations. We start by designing a multi-aspect reward function that aligns with human evaluation of reviews. The aspects are related to the review itself (e.g., criticisms, novelty) and the relationship between the review and the manuscript (i.e., relevance). First, we perform supervised fine-tuning of DeepSeek-R1-Distill-Qwen-7B using LoRA on PeerRT, a new dataset of high-quality top AI conference reviews enriched with reasoning traces. We then apply Group Relative Policy Optimization (GRPO) to train two models: REMOR-H (with the human-aligned reward) and REMOR-U (with a uniform reward). Interestingly, the human-aligned reward penalizes aspects typically associated with strong reviews, leading REMOR-U to produce qualitatively more substantive feedback. Our results show that REMOR-U and REMOR-H achieve more than twice the average rewards of human reviews, non-reasoning state-of-the-art agentic multi-modal AI review systems, and general commercial LLM baselines. We found that while the best AI and human reviews are comparable in quality, REMOR avoids the long tail of low-quality human reviews. We discuss how reasoning is key to achieving these improvements and release the Human-aligned Peer Review Reward (HPRR) function, the Peer Review Reasoning-enriched Traces (PeerRT) dataset, and the REMOR models, which we believe can help spur progress in the area.

Problem

• Improving AI-generated peer reviews to avoid shallow feedback
• Aligning review quality with human standards via multi-objective rewards
• Enhancing review relevance and novelty using reasoning LLMs

Innovation

• Multi-objective reinforcement learning for review generation
• Human-aligned reward function for quality feedback
• Reasoning-enriched dataset for fine-tuning LLMs