Toward Autonomous Long-Horizon Engineering for ML Research

2026-04-14

AI Summary

This work addresses the challenge of sustaining long-horizon machine learning research, which is often hindered by fragmented tasks and discontinuous state management. The authors propose AiScientist, a system that enables multi-day collaborative evolution of specialized agents around a persistent project state through hierarchical task orchestration and a permission-isolated File-as-Bus workspace mechanism. Built upon a “thick state, thin control” paradigm, this design overcomes limitations of conventional dialogue-based handoffs and supports artifact-driven iterative repositioning. Evaluated on PaperBench, AiScientist achieves an average improvement of 10.54 points, and attains an 81.82% Any Medal rate on MLE-Bench Lite. Ablation studies confirm the critical contribution of the File-as-Bus architecture to overall performance.

Technology Category

• Multiagent Systems: Teamwork
• Humans and AI: Planning and Decision Support for Human-Machine Teams
• Machine Learning: Life-Long and Continual Learning

Application Category

• Economics, Online Markets and Human Computation: Research challenges in human and human-AI computation
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Autonomous AI research has advanced rapidly, but long-horizon ML research engineering remains difficult: agents must sustain coherent progress across task comprehension, environment setup, implementation, experimentation, and debugging over hours or days. We introduce AiScientist, a system for autonomous long-horizon engineering for ML research built on a simple principle: strong long-horizon performance requires both structured orchestration and durable state continuity. To this end, AiScientist combines hierarchical orchestration with a permission-scoped File-as-Bus workspace: a top-level Orchestrator maintains stage-level control through concise summaries and a workspace map, while specialized agents repeatedly re-ground on durable artifacts such as analyses, plans, code, and experimental evidence rather than relying primarily on conversational handoffs, yielding thin control over thick state. Across two complementary benchmarks, AiScientist improves PaperBench score by 10.54 points on average over the best matched baseline and achieves 81.82 Any Medal% on MLE-Bench Lite. Ablation studies further show that File-as-Bus protocol is a key driver of performance, reducing PaperBench by 6.41 points and MLE-Bench Lite by 31.82 points when removed. These results suggest that long-horizon ML research engineering is a systems problem of coordinating specialized work over durable project state, rather than a purely local reasoning problem.

Problem

• autonomous AI
• long-horizon engineering
• ML research
• durable state
• task orchestration

Innovation

• autonomous AI research
• long-horizon engineering
• hierarchical orchestration
• File-as-Bus
• durable state continuity