Idea2Plan: Exploring AI-Powered Research Planning

2025-10-28

AI Summary

The capability of large language models (LLMs) to transform ambiguous scientific ideas into structured research plans—a critical yet unexplored aspect of scientific discovery—remains poorly characterized. Method: We introduce the Idea2Plan task and establish Idea2Plan Bench, the first benchmark for scientific planning, comprising 200 carefully selected ICML 2025 papers. We further propose JudgeEval, a comprehensive evaluation framework integrating human-defined rubrics, expert annotations, and automated scoring to quantitatively assess both LLMs’ research planning proficiency and their self-assessment accuracy. Contribution/Results: Experiments reveal that GPT-5 series models achieve the strongest performance, yet substantial gaps persist across key dimensions—including problem decomposition, methodology specification, and feasibility analysis. This work provides the first systematic characterization of LLM capabilities in the upstream phase of scientific discovery, delivering a standardized benchmark and methodological foundation for developing autonomous scientific agents.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have demonstrated significant potential to accelerate scientific discovery as valuable tools for analyzing data, generating hypotheses, and supporting innovative approaches in various scientific fields. In this work, we investigate how LLMs can handle the transition from conceptual research ideas to well-structured research plans. Effective research planning not only supports scientists in advancing their research but also represents a crucial capability for the development of autonomous research agents. Despite its importance, the field lacks a systematic understanding of LLMs' research planning capability. To rigorously measure this capability, we introduce the Idea2Plan task and Idea2Plan Bench, a benchmark built from 200 ICML 2025 Spotlight and Oral papers released after major LLM training cutoffs. Each benchmark instance includes a research idea and a grading rubric capturing the key components of valid plans. We further propose Idea2Plan JudgeEval, a complementary benchmark to assess the reliability of LLM-based judges against expert annotations. Experimental results show that GPT-5 and GPT-5-mini achieve the strongest performance on the benchmark, though substantial headroom remains for future improvement. Our study provides new insights into LLMs' capability for research planning and lay the groundwork for future progress.

Problem

• Investigating LLMs' transition from research ideas to structured plans
• Measuring AI research planning capability through Idea2Plan benchmark
• Assessing reliability of LLM judges against expert annotations

Innovation

• Proposes Idea2Plan task for research planning
• Introduces benchmark from ICML papers with rubrics
• Evaluates LLM judges against expert annotations