To Trust or Not to Trust: Authors' Response to AI-based Reviews

2026-05-15

AI Summary

This study addresses the empirical gap in understanding authors’ experiences with and attitudes toward AI-assisted peer review. Through two independent pilot studies in computer science, combining anonymous surveys, descriptive statistics, and inductive thematic analysis, it systematically examines authors’ perceptions of the utility and credibility of AI-generated reviews and their subsequent revision behaviors. Findings indicate that 83.9% of authors found AI reviews useful, 80.4% acknowledged that AI identified issues overlooked by human reviewers, and 82.1% revised their manuscripts accordingly; however, only a minority expressed full trust in AI, with 76.8% emphasizing the necessity of obtaining explicit consent prior to its use. The study proposes that AI should function as a supervised or author-controlled auxiliary tool, offering empirical evidence and normative guidance for the responsible integration of AI into scholarly peer review.

Abstract

Large language models are increasingly discussed and used as tools that may assist with scholarly peer review, but empirical evidence regarding how authors use and perceive AI-based feedback remains limited. This paper reports findings from two independent pilot studies on authors' use and perceptions of AI-based auxiliary review at two computer science venues. After the review release, authors were invited to complete an anonymous post-review questionnaire about the AI review's usefulness, trustworthiness, agreement with human reviews, practical value for revision, perceived inaccuracies, and consent. The final dataset included 56 analyzable responses from authors of 40 papers; closed-ended items were summarized using descriptive statistics, and open-ended responses were analyzed using inductive thematic analysis. Most respondents (83.9%) considered the AI-based review useful, and 80.4% reported that it identified issues not mentioned by human reviewers. This perceived added value translated into action: 82.1% reported using at least some AI feedback in their camera-ready version. However, the authors did not treat the AI review as equivalent to a human review. They generally trusted it less than the human reviews and found human feedback clearer, even though 25.0% described at least some human reviews as not very useful. Reported problems with the AI review were usually limited: 51.8% reported minor inaccuracies, while 16.1% reported clearly incorrect, misleading, or irrelevant comments. Support for future use was strongest when AI was framed as a supervised or author-controlled tool: 96.4% said they would use AI as an internal review tool before future submissions, 89.3% preferred advance notice that AI would be used in review, and 76.8% favored explicit consent before use.

Problem

• AI-based review
• peer review
• author perception
• trustworthiness
• large language models

Innovation

• AI-assisted peer review
• author perception
• trust in AI
• large language models
• human-AI collaboration