Why Solve It Twice? Hierarchical Accumulation of Skills for Transfer-Efficient ML Engineering

2026-06-29

AI Summary

This work addresses the significant computational inefficiency in current machine learning engineering (MLE) agents, which repeatedly rediscover established techniques from scratch in competitions. To mitigate this redundancy, we propose HASTE, a novel hierarchical skill accumulation system that organizes cross-competition knowledge across three scopes—global, domain-specific, and competition-level—and employs a coordinator to drive large language models in abstracting and transferring skills. Integrating multi-agent collaboration, hierarchical knowledge loading, and controlled ablation validation, HASTE achieves a 77.3% medal rate on MLE-Bench Lite. It reduces optimization iterations by 52% through warm-starting and attains an 85% solution adoption rate when the skill repository exceeds 50 entries, substantially outperforming both flat-loading and no-skill baselines.

Abstract

ML engineering agents waste compute rediscovering known techniques because every competition is a cold start. We present HASTE, a hierarchical multi-agent system that organizes cross-competition knowledge into three scope tiers (global, domain, and competition-specific), each coupled to a matching agent level. An orchestrator coordinates domain specialists and promotes learning between tiers via LLM-driven abstraction. A controlled ablation provides evidence for scoped loading: holding a 159-skill inventory constant across 8 competitions, tiered loading achieves a 100% medal rate while flat loading reaches only 62.5%, the same medal rate as loading no skills, and consumes 2x the output tokens. On the full MLE-Bench Lite benchmark (22 Kaggle competitions), HASTE reaches a medal rate of 77.3% using Claude Sonnet 4.6 at 12h per competition. In a cold-start run, the system begins with no accumulated skills. In warm-start runs, it reloads skills learned from earlier competitions, using only global and domain-level skills for transfer across competitions. Warm starts use 52% fewer refinement iterations, and the fraction of proposed changes kept by the agent rises from 42% at low inventory to 85% once 50+ skills are available. These results suggest that better knowledge organization can partly substitute for model strength and compute budget in ML-engineering agents.

Problem

• ML engineering
• knowledge transfer
• cold start
• skill accumulation
• multi-agent system

Innovation

• Hierarchical Multi-Agent System
• Skill Accumulation
• Transfer-Efficient ML Engineering
• Scoped Knowledge Loading
• LLM-Driven Abstraction