Can LLM design high-quality experiments? A Comprehensive and Systematic Benchmark on Autonomous Experimental Design

2026-08-04

AI Summary

This study addresses the lack of effective evaluation of large language models (LLMs) in systematic experimental design, particularly along two critical dimensions: high-level planning and low-level configuration. To bridge this gap, the authors introduce SCOPE, the first comprehensive benchmark for autonomous experimental design, encompassing 300 top-tier conference papers across 19 domains, which systematically assesses LLM performance in terms of both experimental planning completeness and configuration accuracy. Furthermore, they propose OptED, an agent-based workflow that incorporates stage isolation, tool augmentation, and rule-based constraints to substantially alleviate LLMs’ performance bottlenecks in low-level configuration. Experimental results demonstrate that prevailing LLMs struggle to generate high-quality experimental designs directly, whereas OptED significantly enhances the reasonableness and accuracy of such designs.

Abstract

AI for Research (AI4Research) leverages AI to automate and improve scientific workflows. While experimental design is a critical stage of the research process, prior work has focused primarily on code implementation and execution, overlooking the importance of this stage, and no benchmark exists to evaluate AI's ability to conduct systematic experiment design. To bridge this gap, we propose SCOPE, a Scientific COmprehensive Planning Evaluation Benchmark constructed from 300 high-quality latest papers across 19 research domains from top-tier venues (e.g., ICML, NeurIPS, and ICLR),evaluating LLMs on two dimensions: High-Level planning completeness (main, ablation, and analysis experiments) and Low-Level configuration accuracy and rationality (datasets, baselines, and metrics). Benchmarking reveals three findings: (1) most LLMs cannot directly design high-quality experiments; (2) all LLMs exhibit a performance bottleneck in low-level configuration; and (3) search mode does not improve design quality. Furthermore, to address these challenges, we propose OptED, a novel agentic workflow to optimize LLM-based experimental design, that enhances LLM-based experimental planning through stage isolation, tool augmentation, and rule-based constraints, effectively alleviating the configuration bottleneck.

Problem

• autonomous experimental design
• large language models
• scientific benchmark
• AI for Research
• experimental planning

Innovation

• autonomous experimental design
• large language models
• benchmarking
• AI for Research
• agent-based workflow