AutoData: Agentic Search for Pre-training Data Selection

2026-09-17

AI Summary

本文通过引入AutoData代理，利用执行反馈自动搜索预训练数据选择算法，优化了数据工程，超越了现有的人工策划流程。

Abstract

LLM agents have recently shown promise in automating machine learning engineering by editing model and training code under execution feedback. Data, however, remains largely outside this agentic optimisation loop. We frame pre-training data selection as heuristic engineering over per-document features, i.e., lexical statistics, categorical labels, and perplexity. We introduce AutoData, an agent that searches directly over executable selection algorithms. Unlike prior data mixture methods that optimise weights over a fixed set of domains, AutoData searches a richer program space of scoring, stratification, and stochastic selection rules, discovering feature interactions automatically by iteratively refining algorithms with validation feedback from a proxy model. Within an overnight search, AutoData discovers a selection algorithm that outperforms existing human-designed curation pipelines. Despite being searched only on this small proxy, the discovered recipe transfers to larger scales and improves the downstream metric CORE. These results suggest that data engineering can be treated as an agentic machine learning problem, extending autonomous research from model and training-code optimization to the data.

Problem

• Pre-training Data Selection
• Agentic Search
• Heuristic Engineering
• Feature Interactions

Innovation

• AutoData
• agentic optimization
• pre-training data selection
• executable selection algorithms
• validation feedback