Train Smarter, Not Longer: Memorization-Guided Data Reuse for Efficient LLM Training

2026-07-06

AI Summary

This work addresses a key challenge in large language model training: how to efficiently reuse limited high-quality data to improve sample efficiency while avoiding overfitting. The authors propose a memory-guided adaptive data reuse paradigm that, for the first time, jointly leverages loss retention dynamics and downstream task performance to identify the model’s “memory window.” Based on this window, the method dynamically schedules the timing and frequency of data replay. Breaking away from the conventional empirical limit of no more than four training epochs, the approach demonstrates that—when memory signals are appropriately utilized—model performance can continue to improve with increased data repetition. This provides a novel and theoretically grounded strategy for intelligent and efficient data scheduling in large-scale language model training.

Abstract

The training paradigm of large language models has shifted from traditional one-pass training to multi-epoch training, as reasonable reuse of limited high-quality data can improve both model performance and sample efficiency. Meanwhile, excessive repetition introduces the risk of overfitting and diminishing returns. Determining when and how to reuse data effectively thus emerges as a natural but under-explored question. Through a novel observation of model's "Memorization Window" signals derived from loss retention dynamics and downstream evaluation scores, we propose "Memorization-guided Data Reuse", a training paradigm that adaptively determines when and how data should be reused, enabling principled decisions on the number of training epochs and the scheduling of data replays. Our preliminary experiments reveal a consistent memorization-driven regime: performance continues to improve with repetition far beyond current practice (e.g., the commonly cited four-epoch limit). While a full scheduler remains future work, these insights provide a foundation for memorization-aware training schedules, helping to determine reuse budgets and move toward training LLMs smarter rather than longer with limited high-quality data.

Problem

• data reuse
• overfitting
• training efficiency
• large language models
• memorization

Innovation

• Memorization-Guided Training
• Data Reuse
• Memorization Window
• Multi-Epoch Training
• Sample Efficiency