How Do Agents Fail on AutoResearch: End-to-End Diagnostic Evaluation on 100 Real-World Frontier Research Tasks

2026-08-14

AI Summary

This study addresses the lack of systematic failure diagnosis in existing AutoResearch evaluations by constructing a hundred-task benchmark spanning the entire research lifecycle and proposing a taxonomy comprising 45 distinct failure patterns. Through comprehensive trajectory annotation and an Agent-as-Judge attribution pipeline, we conducted an in-depth analysis of 800 agent trajectories. Our findings reveal that model-level metacognitive deficits constitute the primary bottleneck constraining agent performance. By bridging the gap in understanding failure mechanisms within AutoResearch, this work provides critical theoretical foundations and evaluation standards for enhancing the robustness and self-correction capabilities of scientific research agents.

Abstract

AI has long assisted scientific research, but the rapid advance of LLMs and agentic scaffolds is reshaping the landscape; a single system can now carry whole-stage research from an initial hypothesis all the way to final published paper, which is a paradigm now referred to as AutoResearch. Existing evaluations reveal little about how these agents operate or where they break down. Tasks are narrowly-scoped, evaluation measures performance but not process, and failure diagnoses lack systematic coverage or artifact-level visibility. To address this gap, we introduce AutoResearchEval, featuring 100 tasks grounded in published frontier science across 7 scientific domains and the full research lifecycle, including ideation, retrieval, execution, analysis, writing, and review. Evaluating 8 harness-model combinations yields 800 autoresearch agent trajectories, with process-level annotation. We organize these insights into AutoResearch Failure Taxonomy or ARFT, a framework of 45 empirically-grounded failure patterns. To enable scalable fine-grained attribution, we leverage a human-calibrated agent-as-a-judge pipeline to inspect complete trajectories and intermediate artifacts. Failure patterns converge on a single overarching limitation, namely that current agents lack a metacognitive loop, which entails the ability to check what they produced against what they found, revise when it does not hold up, and question whether the path they took was sound. The same patterns recur across all 8 harness-model combinations, including the strongest models tested, locating the deficit at the model level rather than in any particular scaffold; whether orchestration-level interventions can close it is an open question this work does not test. We publicly release AutoResearchEval and ARFT to facilitate continued research and development in autonomous scientific discovery.

Problem

• AutoResearch
• Agent Failure
• Diagnostic Evaluation
• Failure Taxonomy
• Metacognition

Innovation

• AutoResearchEval
• Failure Taxonomy
• Metacognitive Loop
• Agent-as-a-Judge
• Process-level Annotation