Sem-Detect: Semantic Level Detection of AI Generated Peer-Reviews

2026-05-20

AI Summary

This study addresses the emerging challenge of distinguishing peer reviews authored by humans, generated by large language models (LLMs), or post-edited with LLM assistance—a critical issue for academic integrity. The work introduces argument-level semantic diversity into the detection framework, leveraging the distinctive expressiveness and variability of human reviewers. By comparing semantic similarity between input reviews and AI-generated counterparts produced by multiple models, the method effectively captures nuanced differences in textual features. Evaluated on a large-scale dataset comprising over 20,000 ICLR and NeurIPS reviews, the approach achieves a 25.5% improvement in true positive rate at a 0.1% false positive rate over the strongest baseline in binary classification. In the more challenging three-way classification setting, fewer than 3.5% of LLM-polished reviews are misclassified as purely AI-generated, demonstrating substantially enhanced robustness and practical utility.

Abstract

How can we distinguish whether a peer review was written by a human or generated by an AI model? We argue that, in this setting, authorship should not be attributed solely from the textual features of a review, but also from the ideas, judgments, and claims it expresses. To this end, we propose Sem-Detect, an authorship detection method for peer reviews that operationalizes this principle by combining textual features with claim-level semantic analysis. Sem-Detect compares a target review against multiple AI-generated reviews of the same paper, leveraging the observation that different AI models tend to converge on similar points, while human reviewers introduce more unique and diverse ones. As a result, Sem-Detect is able to distinguish fully AI reviews from authentic human-written ones, including those that have been refined using an LLM but still reflect human judgment. Across a dataset of over 20,000 peer reviews from ICLR and NeurIPS conferences, Sem-Detect improves over the strongest baseline by 25.5% in TPR@0.1% FPR in the binary setting. Moreover, in the three-class scenario, we empirically show that LLM refinement preserves the semantic signals of human reviews, which remain distinct from the patterns exhibited by fully AI-generated text; as a result, fewer than 3.5% of LLM-refined human reviews are misclassified as AI-generated.

Problem

• AI-generated peer reviews
• authorship detection
• semantic analysis
• human vs AI writing
• peer review authenticity

Innovation

• semantic-level detection
• AI-generated peer reviews
• claim-level analysis
• authorship attribution
• LLM-refined reviews