When is Warmstarting Effective for Scaling Language Models?

2026-05-13

AI Summary

This study investigates the limitations of warm-starting language models in scaling training and identifies two key obstacles: an overemphasis on preserving initial performance and a neglect of the interaction between growth factors and hyperparameters. Through systematic experiments—including ablation studies, cross-architecture validation (spanning dense MLPs and language models), scaling law fits, and multi-budget training efficiency comparisons—the work demonstrates that retaining initial performance is unnecessary and that simple, architecture-agnostic growth strategies outperform complex operators. Crucially, the research reveals for the first time an efficiency ceiling for growth factors: 2× scaling most reliably accelerates convergence under low compute budgets (<20 tokens per parameter), beyond which training from scratch becomes more efficient. These findings yield a practical, predictable guideline for effective model scaling.

Abstract

Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited practical adoption in large-scale training. We attribute this to two underexplored factors: (1) an overemphasis on preserving the smaller model's performance at initialization, which constrains operator design for new architectures, and (2) insufficient analysis of how growth interacts with hyperparameters and scaling behavior, compounded by inconsistent growth factors across the literature. We show that preserving the base model's initial post-growth performance is not necessary for strong final performance, and that simple, architecture-agnostic growth strategies can outperform more complex warmstarting operators. Crucially, we empirically identify an upper bound on the growth factor $g$ beyond which training from scratch is more efficient. We observe this across multiple ablation setups. Notably, this limit is also present, but unreported, in prior published results. Across our experiments on dense MLPs and dense language models, we find that a $2\times$ growth factor is the most reliable in yielding convergence speedups, with gains most pronounced under 20 tokens/parameter budgets and diminishing as budget increases. We fit scaling laws over these observations to provide predictive guidance for practitioners deciding when and how much to grow. Together, our analysis provides practical guidelines and empirical limits for model growth.

Problem

• warmstarting
• model scaling
• growth factor
• language models
• training efficiency

Innovation

• warmstarting
• model growth
• scaling laws
• growth factor
• language models