How Far Are We From True Auto-Research?

2026-05-18

AI Summary

This study addresses the lack of systematic evaluation of research papers autonomously generated by AI agents. To this end, it introduces ResearchArena, a framework that guides off-the-shelf agents—such as Claude, GPT, and Kimi—through the complete research pipeline, including topic selection, experimentation, writing, and iterative refinement across 13 computer science tasks. For the first time, the work employs a tripartite evaluation体系 comprising manuscript review, artifact-aware peer review, and human meta-review. Findings reveal that assessments based solely on manuscript appearance substantially overestimate the quality of auto-generated research; none of the 117 generated papers met top-tier conference acceptance standards, with performance sharply declining under artifact evaluation. This exposes three recurring experimental flaws in agent-generated research, underscoring that developing truly capable automated research systems remains a significant challenge.

Abstract

Recent auto-research systems can produce complete papers, but feasibility is not the same as quality, and the field still lacks a systematic study of how good agent-generated papers actually are. We introduce ResearchArena, a minimal scaffold that lets off-the-shelf agents (Claude Code using Opus 4.6, Codex using GPT-5.4, and Kimi Code using K2.5) carry out the full research loop themselves (ideation, experimentation, paper writing, self-refinement) under only lightweight guidance. Across 13 computer science seeds and 3 trials per agent-domain pair, ResearchArena yields 117 agent-generated papers, each evaluated under three complementary lenses: a manuscript-only reviewer (SAR), an artifact-aware peer review (PR) in which agents inspect the workspace alongside the manuscript, and an human conducted meta-review. Under SAR alone the picture is optimistic: Claude Code obtains the highest score, outperforms Analemma's FARS, and matches the weighted-average human ICLR 2025 submission, suggesting that minimally scaffolded agents can produce papers that look competitive on manuscript-only review. Manual inspection, however, reveals this picture is overstated: SAR scores are poorly aligned with its actual acceptance decisions and reward plausible framing without verifying experimental substance. Under artifact-aware PR scores drop sharply, and manual auditing identifies experimental rigor as the major bottleneck, decomposing into three failure modes (fabricated results, underpowered experiments, and plan/execution mismatch) that are highly agent-dependent: Codex 5%/8% paper-vs-artifact mismatch / fabricated references versus Kimi Code 77%/72%, a $\sim$15$\times$ spread that tracks distinct research personas the agents develop. None of the 117 agent-generated papers reaches the acceptance bar of a top-tier venue. This suggests that we are still gapped from the true auto-research.

Problem

• auto-research
• agent-generated papers
• experimental rigor
• artifact-aware review
• research quality evaluation

Innovation

• auto-research
• ResearchArena
• agent-generated papers
• artifact-aware review
• experimental rigor