Towards Next-Generation LLM Training: From the Data-Centric Perspective

2026-03-15

AI Summary

This work addresses the inefficiency and lack of dynamic optimization in data preparation for large language model (LLM) training. To overcome these limitations, we propose a novel end-to-end, agent-driven paradigm for adaptive data utilization, introducing intelligent agents into the data preparation pipeline for the first time. Our approach enables dynamic data selection, mixing, and reweighting during training through an automated, agent-based workflow, while establishing a unified framework for data-model interaction. By replacing conventional static data processing with this adaptive mechanism, our method significantly reduces manual intervention, enhances training efficiency and robustness, and offers a scalable, reusable, data-centric solution for next-generation LLMs.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Large language models (LLMs) have demonstrated remarkable performance across a wide range of tasks and domains, with data playing a central role in enabling these advances. Despite this success, the preparation and effective utilization of the massive datasets required for LLM training remain major bottlenecks. In current practice, LLM training data is often constructed using ad hoc scripts, and there is still a lack of mature, agent-based data preparation systems that can automatically construct robust and reusable data workflows, thereby freeing data scientists from repetitive and error-prone engineering efforts. Moreover, once collected, datasets are often consumed largely in their entirety during training, without systematic mechanisms for data selection, mixture optimization, or reweighting. To address these limitations, we advocate two complementary research directions. First, we propose building a robust, agent-based automatic data preparation system that supports automated workflow construction and scalable data management. Second, we argue for a unified data-model interaction training system in which data is dynamically selected, mixed, and reweighted throughout the training process, enabling more efficient, adaptive, and performance-aware data utilization. Finally, we discuss the remaining challenges and outline promising directions for future research and system development.

Problem

• large language models
• data preparation
• data utilization
• training data
• data-centric AI

Innovation

• agent-based data preparation
• dynamic data selection
• data-model interaction
• automated workflow construction
• data mixture optimization