E3: Issue-Level Backtesting for Automated Research Critique

2026-05-26

AI Summary

This work addresses the challenge that common technical flaws in scientific papers—such as unsupported claims, missing ablation studies, and weak baselines—are often overlooked by human reviewers. To tackle this issue, the authors propose E3, an automated peer-review assistant powered by large language models. E3 introduces a novel problem-level regression testing framework that, without contaminating training data, systematically classifies issues, analyzes supporting evidence, and incorporates an anonymous meta-review mechanism to automatically detect, localize, and suggest remedies for defects. Evaluated on 100 ICLR 2026 submissions, E3 achieves a permissive recall of 90.2% and a strict recall of 65.8%, substantially outperforming both GPT and Claude models as well as human reviewers, while uncovering an additional 1,635 previously missed issues.

Abstract

We present E3, an automated review assistant that augments reviewers and engineering teams by identifying decision-relevant technical concerns in research papers. For each concern, E3 reports its nature, its location, its bearing on the contribution, and the analysis or evidence that would resolve it, covering unsupported claims, missing ablations, weak baselines, hidden assumptions, threats to validity, and leakage risks. To evaluate E3 without contamination confounds we adopt an issue-level backtesting protocol: the corpus is restricted to papers postdating the training cutoff of every automated source, and for each paper a meta-judge that observes only anonymised reviews labels every issue-source pair as Caught, Partial, or Missed. Applied to 100 ICLR 2026 papers and 4598 judged issue rows, comparing E3 against the ICLR human reviews and two prompt-matched LLM baselines built on gpt-5.4 from OpenAI and claude-opus-4-6 from Anthropic, with meta-judge gpt-5.5, E3 attains the highest recall on every aggregate metric. Partial-inclusive recall reaches 90.2 percent, which is 15.5 points over GPT, 17.1 points over Claude, and 29.2 points over the human reviews, and strict recall preserves the ordering at 65.8 percent. On concerns raised by the human reviewers, E3 recovers 89.6 percent; on concerns the human reviewers missed it surfaces 1635 additional rows admitted into the judged union, 406 above the next-best source. Corpus, baseline prompts, judge prompt template, and evaluation code are released.

Problem

• automated research critique
• technical concerns
• peer review
• issue detection
• scientific paper evaluation

Innovation

• issue-level backtesting
• automated research critique
• technical concern detection
• recall evaluation
• review augmentation