ResearchGym: Evaluating Language Model Agents on Real-World AI Research

2026-02-16

AI Summary

This work addresses the lack of systematic benchmarks for evaluating AI agents’ capabilities in end-to-end, real-world scientific research tasks. We introduce ResearchGym, the first environment that transforms authentic scientific workflows into containerized, reproducible evaluation tasks, built upon five top-tier conference papers and encompassing hypothesis generation, experimental design, and performance surpassing. Through closed-loop evaluations with state-of-the-art models—including GPT-5 and Claude Code (Opus-4.5)—we find that agents complete only 26.5% of subtasks on average and exceed human baselines by 11.5% in just 1 out of 15 attempts (6.7%). Notably, one agent successfully reproduced an ICML 2025 Spotlight result, highlighting both the potential and the reliability gap of current large models in long-horizon scientific discovery.

Technology Category

• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

We introduce ResearchGym, a benchmark and execution environment for evaluating AI agents on end-to-end research. To instantiate this, we repurpose five oral and spotlight papers from ICML, ICLR, and ACL. From each paper's repository, we preserve the datasets, evaluation harness, and baseline implementations but withhold the paper's proposed method. This results in five containerized task environments comprising 39 sub-tasks in total. Within each environment, agents must propose novel hypotheses, run experiments, and attempt to surpass strong human baselines on the paper's metrics. In a controlled evaluation of an agent powered by GPT-5, we observe a sharp capability--reliability gap. The agent improves over the provided baselines from the repository in just 1 of 15 evaluations (6.7%) by 11.5%, and completes only 26.5% of sub-tasks on average. We identify recurring long-horizon failure modes, including impatience, poor time and resource management, overconfidence in weak hypotheses, difficulty coordinating parallel experiments, and hard limits from context length. Yet in a single run, the agent surpasses the solution of an ICML 2025 Spotlight task, indicating that frontier agents can occasionally reach state-of-the-art performance, but do so unreliably. We additionally evaluate proprietary agent scaffolds including Claude Code (Opus-4.5) and Codex (GPT-5.2) which display a similar gap. ResearchGym provides infrastructure for systematic evaluation and analysis of autonomous agents on closed-loop research.

Problem

• AI research agents
• benchmarking
• autonomous research
• language model evaluation
• scientific discovery

Innovation

• ResearchGym
• AI research agents
• autonomous scientific discovery
• long-horizon evaluation
• agent reliability gap