PeerCheck: Enhancing LLM-Generated Academic Reviews Towards Human-Level Quality

2026-06-18

AI Summary

This work addresses the dual challenges of surging academic submissions and the subpar review quality of large language models (LLMs) compared to human reviewers by introducing the PeerCheck framework. Through fine-grained comparative analysis, the study uncovers systematic discrepancies between LLM-generated and human reviews in terms of focus and emphasis. To bridge this gap, the authors integrate Chain-of-Thought prompting with Retrieval-Augmented Generation (RAG), demonstrating that Chain-of-Thought significantly enhances review quality. Notably, the research identifies a previously unreported “RAG paradox” in peer review, wherein RAG’s benefits are counteracted under certain conditions. The proposed approach yields LLM-generated reviews that better align with human reviewers’ focal points and stylistic nuances. To foster further research, the authors publicly release their dataset.

Abstract

As academic submissions grow, the traditional peer review process struggles to keep up, raising concerns about quality and fairness. A trend of using large language models (LLMs) for assistance has emerged. In this work, we take a critical step toward improving the quality of LLM-generated reviews. We propose the PeerCheck framework, which investigates LLM-human review differences (RQ1) and explores methods to improve LLM-generated review quality (RQ2). We first analyzed the human-written reviews with reviews generated by various LLMs and found that LLMs and humans focus on different terms, e.g., LLMs prioritize theory while humans emphasize methodology and experiments. We further adopt prompt engineering, such as Chain-of-Thought (CoT), and utilize retrieval-augmented generation (RAG) to enhance the LLM-generated reviews towards human-level quality. We find CoT significantly improves the quality of LLM reviews, while we discover an unexpected "RAG paradox," i.e., experiments with RAG produce different results for various LLMs and, in some cases, even reduce review quality. Our comprehensive analysis of LLM-generated academic reviews illustrates both possibilities and limitations, contributing to a more effective, human-aligned review system. Our dataset is available on https://github.com/TrustAIRLab/PeerCheck.

Problem

• LLM-generated reviews
• peer review
• review quality
• human-level quality
• academic reviewing

Innovation

• PeerCheck
• Chain-of-Thought
• Retrieval-Augmented Generation
• LLM-generated reviews
• RAG paradox