Benchmarking Agentic Review Systems

2026-06-17

AI Summary

This work addresses the lack of systematic evaluation frameworks for AI peer-review systems, which hinders reliable assessment of their effectiveness and robustness in scholarly reviewing. The authors propose the first dual benchmark combining external quality signals with controllable error injection to construct a multidimensional evaluation framework. They evaluate six state-of-the-art large language models—including GPT-5.5—on real submissions from ICLR and NeurIPS as well as perturbed arXiv papers, integrating systems such as OpenAIReview, coarse, and Reviewer3. The best-performing configuration (OpenAIReview + GPT-5.5) achieves 83.0% accuracy in paper quality discrimination and 71.6% recall in error detection; combining multiple models further improves error detection recall to 83.3%. Positive user feedback confirms the system’s practical utility and highlights complementary strengths among models.

Abstract

A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated. We evaluate two open-source systems (OpenAIReview and coarse), one proprietary system (Reviewer3), and a zero-shot baseline, across six LLMs spanning frontier and efficient models. First, we study whether AI reviews on ICLR/NeurIPS papers track with papers' quality as approximated by external signals such as citations and acceptance decisions. Every system performs above chance in pairwise accuracy, and the best is OpenAIReview + GPT-5.5 at 83.0%. Second, to test whether systems can catch errors with known ground truth, we construct a perturbation benchmark that injects four categories of errors into papers across eight arXiv subject classes and measure detection recall. The strongest configuration (OpenAIReview + GPT-5.5) catches 71.6% of injected errors, leaving substantial room for improvement. The union of detections across six models reaches 83.3% recall, suggesting different models detect different errors and better harness design can potentially increase performance. Beyond these benchmarks, we study a public deployment of OpenAIReview with real users. Votes on its comments skew positive at 1.44 to 1, and the most common complaints are about false positives and minor nitpicks. Together, by evaluating full review systems backed by state-of-the-art models on real research papers, we show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important errors, and earn positive feedback from real users.

Problem

• agentic review systems
• peer review
• AI-assisted research
• error detection
• benchmarking

Innovation

• agentic review systems
• perturbation benchmark
• AI peer review
• error detection
• LLM evaluation