Pre-training under infinite compute

2025-09-18

AI Summary

Under the prevailing pretraining paradigm—characterized by scarce web text data but abundant computational resources—the key bottleneck lies in data efficiency. Method: This work proposes a synergistic optimization framework integrating strong weight decay regularization, multi-stage progressive training, parameter-scale expansion, and ensemble knowledge distillation. Performance bounds are modeled via power-law scaling laws to systematically lower the asymptotic loss floor. Results: The approach achieves superior performance using only 200M tokens—outperforming baseline models while reducing data requirements by 5.17×. Lightweight student models retain 83% of the ensemble’s gains, and downstream tasks demonstrate an average 17.5× improvement in data efficiency. This study provides the first systematic empirical validation of algorithm-driven, data-efficient pretraining under the “small data + large compute” paradigm, establishing a novel framework for resource-constrained settings.

Technology Category

• Natural Language Processing: Learning & Optimization for NLP
• Machine Learning: Optimization
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We first show that existing data-constrained approaches of increasing epoch count and parameter count eventually overfit, and we significantly improve upon such recipes by properly tuning regularization, finding that the optimal weight decay is $30 imes$ larger than standard practice. Since our regularized recipe monotonically decreases loss following a simple power law in parameter count, we estimate its best possible performance via the asymptote of its scaling law rather than the performance at a fixed compute budget. We then identify that ensembling independently trained models achieves a significantly lower loss asymptote than the regularized recipe. Our best intervention combining epoching, regularization, parameter scaling, and ensemble scaling achieves an asymptote at 200M tokens using $5.17 imes$ less data than our baseline, and our data scaling laws predict that this improvement persists at higher token budgets. We find that our data efficiency gains can be realized at much smaller parameter counts as we can distill an ensemble into a student model that is 8$ imes$ smaller and retains $83%$ of the ensembling benefit. Finally, our interventions designed for validation loss generalize to downstream benchmarks, achieving a $9%$ improvement for pre-training evals and a $17.5 imes$ data efficiency improvement over continued pre-training on math mid-training data. Our results show that simple algorithmic improvements can enable significantly more data-efficient pre-training in a compute-rich future.

Problem

• Optimizing pre-training with fixed data and unlimited compute
• Overcoming overfitting in data-constrained language model training
• Improving data efficiency through regularization and ensembling techniques

Innovation

• Optimizing regularization with large weight decay
• Employing ensemble scaling for lower loss asymptote
• Distilling ensemble into smaller student model