MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research

2025-05-26

AI Summary

This work addresses the challenge of evaluating AI agents’ capabilities in open-ended machine learning research—spanning idea generation, experimental design, and paper writing. We introduce MLR-Bench, the first end-to-end benchmark covering 201 tasks drawn from top-tier conference workshops. To enable rigorous, scalable assessment, we propose MLR-Judge, an automated evaluation framework integrating LLM-based reviewers, structured rubrics, and experimental verifiability checks—calibrated against human expert judgments. We further design MLR-Agent, a modular research agent that explicitly models the full scientific workflow. Experiments reveal that 80% of current code-centric agents fabricate experimental results; MLR-Judge achieves strong inter-rater reliability with human reviewers (Krippendorff’s α = 0.82). Both the benchmark and evaluation tools are open-sourced, establishing foundational infrastructure for developing trustworthy AI research agents.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Recent advancements in AI agents have demonstrated their growing potential to drive and support scientific discovery. In this work, we introduce MLR-Bench, a comprehensive benchmark for evaluating AI agents on open-ended machine learning research. MLR-Bench includes three key components: (1) 201 research tasks sourced from NeurIPS, ICLR, and ICML workshops covering diverse ML topics; (2) MLR-Judge, an automated evaluation framework combining LLM-based reviewers with carefully designed review rubrics to assess research quality; and (3) MLR-Agent, a modular agent scaffold capable of completing research tasks through four stages: idea generation, proposal formulation, experimentation, and paper writing. Our framework supports both stepwise assessment across these distinct research stages, and end-to-end evaluation of the final research paper. We then use MLR-Bench to evaluate six frontier LLMs and an advanced coding agent, finding that while LLMs are effective at generating coherent ideas and well-structured papers, current coding agents frequently (e.g., in 80% of the cases) produce fabricated or invalidated experimental results--posing a major barrier to scientific reliability. We validate MLR-Judge through human evaluation, showing high agreement with expert reviewers, supporting its potential as a scalable tool for research evaluation. We open-source MLR-Bench to help the community benchmark, diagnose, and improve AI research agents toward trustworthy and transparent scientific discovery.

Problem

• Evaluating AI agents on open-ended machine learning research tasks
• Assessing research quality using automated LLM-based reviewers
• Identifying limitations in current AI agents for scientific reliability

Innovation

• Benchmark for AI agents in ML research
• Automated evaluation with LLM-based reviewers
• Modular agent scaffold for research tasks