KompeteAI: Accelerated Autonomous Multi-Agent System for End-to-End Pipeline Generation for Machine Learning Problems

2025-08-13

AI Summary

Existing LLM-driven AutoML systems suffer from two key bottlenecks: (1) rigid search strategies—single-shot generation lacks diversity, and Monte Carlo Tree Search (MCTS) fails to effectively reuse high-quality partial solutions; and (2) low execution efficiency—lengthy code validation cycles impede iterative optimization. This paper proposes an accelerated multi-agent AutoML framework. Its core innovations include: (i) a solution fusion mechanism that recomposes locally optimal sub-solutions; (ii) retrieval-augmented generation (RAG) grounded in real-world Kaggle and arXiv case studies; and (iii) a predictive scoring model coupled with a lightweight debugger to enable early termination and rapid validation. On MLE-Bench, our method achieves a 3% average performance gain over baselines including RD-Agent and AIDE, with a 6.9× speedup in evaluation. On the new Kompete-Bench benchmark, it establishes a new state-of-the-art.

Technology Category

• Machine Learning: Auto ML and Hyperparameter Tuning
• Search and Optimization: Learning to Search
• Multiagent Systems: Multiagent Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent Large Language Model (LLM)-based AutoML systems demonstrate impressive capabilities but face significant limitations such as constrained exploration strategies and a severe execution bottleneck. Exploration is hindered by one-shot methods lacking diversity and Monte Carlo Tree Search (MCTS) approaches that fail to recombine strong partial solutions. The execution bottleneck arises from lengthy code validation cycles that stifle iterative refinement. To overcome these challenges, we introduce KompeteAI, a novel AutoML framework with dynamic solution space exploration. Unlike previous MCTS methods that treat ideas in isolation, KompeteAI introduces a merging stage that composes top candidates. We further expand the hypothesis space by integrating Retrieval-Augmented Generation (RAG), sourcing ideas from Kaggle notebooks and arXiv papers to incorporate real-world strategies. KompeteAI also addresses the execution bottleneck via a predictive scoring model and an accelerated debugging method, assessing solution potential using early stage metrics to avoid costly full-code execution. This approach accelerates pipeline evaluation 6.9 times. KompeteAI outperforms leading methods (e.g., RD-agent, AIDE, and Ml-Master) by an average of 3% on the primary AutoML benchmark, MLE-Bench. Additionally, we propose Kompete-bench to address limitations in MLE-Bench, where KompeteAI also achieves state-of-the-art results

Problem

• Overcoming constrained exploration in AutoML systems
• Addressing execution bottlenecks in pipeline generation
• Enhancing solution diversity with dynamic exploration

Innovation

• Dynamic solution space exploration with merging stage
• Retrieval-Augmented Generation from Kaggle and arXiv
• Predictive scoring and accelerated debugging method