Scaling Domain Data Repetition in LLM Pretraining

2026-08-14

AI Summary

This study addresses the challenges of scarce high-quality domain data and overfitting caused by data reuse in large language model pretraining. We reveal a scaling law demonstrating that the optimal number of data repetitions increases marginally with model size and is negatively correlated with validation loss. Leveraging this insight, we propose a novel paradigm that utilizes small proxy models to estimate optimal repetition counts for large models, thereby circumventing prohibitive full-scale tuning costs. Empirical results confirm that this approach accurately guides data mixing ratios and significantly mitigates the dilution of high-quality data. Consequently, this work provides both a theoretical foundation and an efficient practical framework for data reuse strategies in large-scale pretraining, enabling more effective utilization of limited domain-specific datasets without compromising generalization performance.

Abstract

As large language models scale, their training-token budgets must also increase to maintain an appropriate tokens-per-parameter ratio (\(\mathrm{TPP}\)). However, high-quality domain data is much harder to scale than general web data. As model size and the training-token budget increase, its fraction in the training mixture tends to decrease. Repeating the available high-quality data provides an effective way to counteract this dilution, but excessive repetition may lead to overfitting. We study this trade-off under practical LLM scaling, where the training-token budget grows proportionally with model size. For a fixed domain, we first find that, surprisingly at a fixed \(\mathrm{TPP}\), the optimal repetition count mildly increases with model size. Across different domains, we find that the optimal repetition count is strongly negatively correlated with the final validation loss of a domain: domains with lower loss can generally benefit from more repetitions. In contrast, the amount of unique domain data is only weakly related to the optimal repetition count. These findings suggest that repetition counts tuned on smaller proxy models with the same \(\mathrm{TPP}\) can provide a practical estimate for larger models.

Problem

• LLM pretraining
• data repetition
• scaling laws
• overfitting
• domain data

Innovation

• Data Repetition
• Scaling Laws
• Tokens-per-Parameter
• Proxy Model Tuning
• Domain Data Mixing