Understanding and Supporting Peer Review Using AI-reframed Positive Summary

2025-03-13

AI Summary

This study addresses the problem that critical peer-review feedback often undermines author motivation. We propose a novel “AI-reframed positive summary” paradigm: leveraging NLP and sentiment rewriting models to automatically transform harsh reviews into constructive, encouraging summaries. Through a two-factor experiment (review overall evaluation level × presence/absence of AI-reframed summary), combined with experimental psychology design and linear mixed-effects modeling, we find that this paradigm significantly improves authors’ acceptance of negative feedback—particularly increasing revision effort in the low-evaluation condition. Our work provides the first empirical validation that AI-driven psychological reframing of feedback can simultaneously preserve review validity and enhance author psychological acceptability, thereby improving the sustainability and collaborativeness of the peer-review process. These findings offer both theoretical foundations and practical implementation pathways for intelligent scholarly support systems.

Technology Category

• Natural Language Processing: Summarization
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

While peer review enhances writing and research quality, harsh feedback can frustrate and demotivate authors. Hence, it is essential to explore how critiques should be delivered to motivate authors and enable them to keep iterating their work. In this study, we explored the impact of appending an automatically generated positive summary to the peer reviews of a writing task, alongside varying levels of overall evaluations (high vs. low), on authors' feedback reception, revision outcomes, and motivation to revise. Through a 2x2 online experiment with 137 participants, we found that adding an AI-reframed positive summary to otherwise harsh feedback increased authors' critique acceptance, whereas low overall evaluations of their work led to increased revision efforts. We discuss the implications of using AI in peer feedback, focusing on how AI-driven critiques can influence critique acceptance and support research communities in fostering productive and friendly peer feedback practices.

Problem

• Impact of AI-generated positive summaries on peer review feedback reception.
• Effect of varying evaluation levels on authors' revision efforts and motivation.
• Role of AI in enhancing critique acceptance and fostering friendly peer feedback.

Innovation

• AI-generated positive summaries enhance feedback reception
• Positive summaries increase critique acceptance in peer reviews
• AI-driven critiques foster productive peer feedback practices