AutoResearchBench: Benchmarking AI Agents on Complex Scientific Literature Discovery

2026-04-28

AI Summary

This work addresses the challenge of evaluating AI agents’ capabilities in discovering complex scientific literature by introducing AutoResearchBench, the first benchmark specifically designed for research-oriented scenarios. The benchmark comprises two task types: Deep Research, which requires multi-step tracing to locate target papers, and Wide Research, which demands comprehensive collection of relevant publications. It emphasizes fine-grained comprehension of bibliographic details, open-ended answer sets, and adaptive search strategies, distinguishing it markedly from general-purpose web-browsing benchmarks. Experimental results reveal that even the most advanced large language models achieve only 9.39% accuracy on Deep Research tasks and an Intersection-over-Union (IoU) of 9.31% on Wide Research tasks, underscoring the benchmark’s difficulty. The dataset and evaluation code have been publicly released.

Abstract

Autonomous scientific research is significantly advanced thanks to the development of AI agents. One key step in this process is finding the right scientific literature, whether to explore existing knowledge for a research problem, or to acquire evidence for verifying assumptions and supporting claims. To assess AI agents' capability in driving this process, we present AutoResearchBench, a dedicated benchmark for autonomous scientific literature discovery. AutoResearchBench consists of two complementary task types: (1) Deep Research, which requires tracking down a specific target paper through a progressive, multi-step probing process, and (2) Wide Research, which requires comprehensively collecting a set of papers satisfying given conditions. Compared to previous benchmarks on agentic web browsing, AutoResearchBench is distinguished along three dimensions: it is research-oriented, calling for in-depth comprehension of scientific concepts; literature-focused, demanding fine-grained utilization of detailed information; and open-ended, involving an unknown number of qualified papers and thus requiring deliberate reasoning and search throughout. These properties make AutoResearchBench uniquely suited for evaluating autonomous research capabilities, and extraordinarily challenging. Even the most powerful LLMs, despite having largely conquered general agentic web-browsing benchmarks such as BrowseComp, achieve only 9.39% accuracy on Deep Research and 9.31% IoU on Wide Research, while many other strong baselines fall below 5%. We publicly release the dataset and evaluation pipeline to facilitate future research in this direction. We publicly release the dataset, evaluation pipeline, and code at https://github.com/CherYou/AutoResearchBench.

Problem

• scientific literature discovery
• AI agents
• autonomous research
• benchmarking
• open-ended retrieval

Innovation

• scientific literature discovery
• AI agents
• benchmark
• autonomous research
• open-ended reasoning