Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning

2025-06-12

AI Summary

This work systematically investigates the efficacy and underlying mechanisms of curriculum learning in large language model (LLM) pretraining. Addressing the lack of rigorous empirical validation and a universally applicable difficulty metric in prior work, we propose a multidimensional data difficulty modeling framework grounded in six linguistic and information-theoretic measures—including compression ratio, lexical diversity, and readability. We further design a plug-and-play curriculum warmup strategy that integrates pacing functions with interleaved multi-stage scheduling, enabling efficient curriculum pretraining atop standard Transformer architectures. Extensive experiments across eight benchmarks demonstrate that our approach significantly accelerates convergence during early-to-mid training stages, yields up to a 3.5% performance gain during warmup, and markedly improves data efficiency and generalization. Crucially, this is the first empirical study to establish curriculum learning as a broadly applicable optimization paradigm for LLM pretraining.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Curriculum learning has shown promise in improving training efficiency and generalization in various machine learning domains, yet its potential in pretraining language models remains underexplored, prompting our work as the first systematic investigation in this area. We experimented with different settings, including vanilla curriculum learning, pacing-based sampling, and interleaved curricula-guided by six difficulty metrics spanning linguistic and information-theoretic perspectives. We train models under these settings and evaluate their performance on eight diverse benchmarks. Our experiments reveal that curriculum learning consistently improves convergence in early and mid-training phases, and can yield lasting gains when used as a warmup strategy with up to $3.5%$ improvement. Notably, we identify compression ratio, lexical diversity, and readability as effective difficulty signals across settings. Our findings highlight the importance of data ordering in large-scale pretraining and provide actionable insights for scalable, data-efficient model development under realistic training scenarios.

Problem

• Investigates curriculum learning for efficient language model pretraining
• Evaluates six difficulty metrics to guide data ordering strategies
• Demonstrates curriculum learning improves convergence and model performance

Innovation

• Curriculum learning for efficient pretraining
• Six difficulty metrics guide training
• Compression ratio as key signal