AgentIdeaBench: Benchmarking Scientific Ideation in the Agent Era

2026-09-07

AI Summary

为解决现有评估方法不能准确衡量AI科学家科学构思能力的问题，本文提出AgentIdeaBench基准，通过静态观察和主动探索两种设置来评价33个语言模型在科学构思上的表现。

Abstract

Scientific ideation is the capacity to formulate novel and testable hypotheses from scientific evidence, and autonomous AI scientists depend on it. Existing evaluations largely assess it by asking models to generate ideas from a static, curated set of reference papers. That passive setup departs from the retrieval-and-reasoning workflow of modern AI scientists, and it becomes less discriminative as models improve. We introduce AgentIdeaBench, a multidisciplinary benchmark that evaluates scientific ideation under two matched settings, static observation and active exploration. We report matched Static-Active evaluations for 33 LLMs across 40 densely scored subfields spanning five disciplines, using a multidimensional, literature-verified scoring framework whose critics assess originality against retrieved prior art. Active exploration reveals considerably more capability headroom, and that headroom is unevenly distributed across models. Performance scales about twice as fast as under static observation, and the exploration gain is capability-gated, favoring the strongest models over the weakest. The gain reflects better grounding, improving feasibility, clarity, and specificity while leaving measured originality unchanged under our critics. We further explore Scientific World Modeling, a generation-time loop that refines a draft hypothesis through structured thought experiments. It benefits mid-capability models, and its impact diminishes among frontier models that appear to have internalized such reasoning patterns already. AgentIdeaBench gives future work on scientific ideation a measurement basis suited to the agent era.

Problem

• Scientific Ideation
• Autonomous AI Scientists
• Static Observation
• Active Exploration
• Retrieval-and-Reasoning Workflow

Innovation

• Scientific Ideation
• Active Exploration
• Static Observation
• Scientific World Modeling
• Capability Gating