MLReplicate: Benchmarking Autonomous Research Systems for Machine Learning Reproducibility

2026-05-15

AI Summary

This work addresses the absence of end-to-end, reliable evaluation frameworks for autonomous scientific research systems in the context of machine learning reproducibility. It introduces MLReplicate, the first standardized benchmark for this task, constructed from outstanding papers accepted at ICML 2025. The authors employ a dual-track evaluation protocol combining automated conference-style peer review with structured expert assessment to systematically evaluate six state-of-the-art systems on their ability to generate complete scientific manuscripts. Results reveal that only 10 out of 45 generated manuscripts passed automated review, and 59% of those deemed acceptable contained fabricated or unsupported claims. Notably, workflow design proved more critical than computational scale: the system with the lowest resource consumption outperformed the highest-cost system in human evaluations despite a 38-fold difference in token usage, exposing widespread methodological flaws and hallucination in current approaches.

Abstract

Autonomous research systems capable of generating complete scientific manuscripts have advanced rapidly, yet robust and realistic evaluation frameworks have failed to keep pace. To bridge this gap, we introduce MLReplicate, an end-to-end benchmark evaluating autonomous research systems on machine learning reproducibility. The benchmark was constructed from ICML 2025 outstanding papers reformulated into standardized input specifications and evaluated across 6 state-of-the-art research systems: AI SCIENTIST-V1, AI SCIENTIST-V2, AGENT LABORATORY, CYCLERESEARCHER, AI RESEARCHER, and TINY SCIENTIST, yielding 45 generated manuscripts, with 3 failed experiments. Outputs are assessed using a dual-protocol approach that combines automated conference-style review and structured expert human evaluation, while tracking computational cost, runtime, and the amount of required human intervention. The automated conference-style review accepted 10 out of 37 valid submissions. An additional 8 submissions were desk-rejected before review for failing to meet the minimum page threshold. In contrast to automated reviews, human reviewers consistently identified methodological flaws, hallucinated experimental results, and reproducibility failures across all systems, and 59% of accepted automated reviews contained fabricated or unsupported claims. We further find that neither token budget nor computational cost predicts output quality: the cheapest system outperforms the most resource-intensive system in human evaluation, despite a 38-fold difference in input tokens. We thus demonstrate that autonomous research workflow design matters more than the scale of compute. MLReplicate exposes a substantial gap between current autonomous research systems and genuine scientific rigor, and establishes a practical, extensible evaluation framework for systematic progress toward trustworthy AI-driven scientific discovery.

Problem

• autonomous research systems
• machine learning reproducibility
• evaluation benchmark
• scientific rigor
• AI-driven scientific discovery

Innovation

• autonomous research systems
• reproducibility benchmark
• AI-generated science
• dual-protocol evaluation
• scientific rigor