InnovatorBench: Evaluating Agents' Ability to Conduct Innovative LLM Research

2025-10-31

AI Summary

Existing benchmarks inadequately evaluate AI agents’ capacity for end-to-end scientific exploration—encompassing hypothesis generation, experimental design, executable code implementation, and analysis—in LLM-driven research. Method: We introduce the first research-capability-oriented benchmark, covering 20 tasks across six categories (e.g., data construction, loss-function design), requiring agents to generate runnable code and be evaluated holistically on correctness, performance, and output quality. We develop ResearchGym—a scalable platform supporting long-horizon, distributed execution and asynchronous monitoring—and design integrative tasks demanding tangible scientific outputs. Leveraging the ReAct framework, we integrate state-of-the-art models (e.g., Claude-4, GPT-5) for executable planning and snapshot-based checkpointing. Results: Experiments reveal that while current models show promise on coding tasks, they exhibit poor robustness and weak resource management in algorithm-sensitive and long-horizon decision-making scenarios, averaging over 11 hours to converge to optimal solutions—demonstrating the benchmark’s rigor and scientific utility.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions

Abstract

AI agents could accelerate scientific discovery by automating hypothesis formation, experiment design, coding, execution, and analysis, yet existing benchmarks probe narrow skills in simplified settings. To address this gap, we introduce InnovatorBench, a benchmark-platform pair for realistic, end-to-end assessment of agents performing Large Language Model (LLM) research. It comprises 20 tasks spanning Data Construction, Filtering, Augmentation, Loss Design, Reward Design, and Scaffold Construction, which require runnable artifacts and assessment of correctness, performance, output quality, and uncertainty. To support agent operation, we develop ResearchGym, a research environment offering rich action spaces, distributed and long-horizon execution, asynchronous monitoring, and snapshot saving. We also implement a lightweight ReAct agent that couples explicit reasoning with executable planning using frontier models such as Claude-4, GPT-5, GLM-4.5, and Kimi-K2. Our experiments demonstrate that while frontier models show promise in code-driven research tasks, they struggle with fragile algorithm-related tasks and long-horizon decision making, such as impatience, poor resource management, and overreliance on template-based reasoning. Furthermore, agents require over 11 hours to achieve their best performance on InnovatorBench, underscoring the benchmark's difficulty and showing the potential of InnovatorBench to be the next generation of code-based research benchmark.

Problem

• Evaluating AI agents' ability to automate LLM research tasks
• Assessing end-to-end performance in realistic research environments
• Identifying limitations in long-horizon decision making and algorithm design

Innovation

• Developed InnovatorBench benchmark for realistic LLM research assessment
• Created ResearchGym environment with distributed execution and monitoring
• Implemented ReAct agent combining reasoning with executable planning