Progressive Residual Warmup for Language Model Pretraining

2026-03-05

AI Summary

This work proposes Progressive Residual Warmup (ProRes) to address the challenges of training instability and slow convergence in large language model pretraining. ProRes progressively activates residual connections layer by layer during early training, implementing a “shallow-first” learning mechanism that enables gradual, layer-wise residual scaling. This approach steers optimization along a more favorable trajectory, enhancing convergence behavior. Built upon the Transformer architecture, ProRes integrates a hierarchical residual scaling strategy with tailored normalization and initialization schemes. Experiments across multiple model scales demonstrate that ProRes significantly improves training stability, accelerates convergence, and boosts both generalization capability and downstream task performance.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Web Mining and Content Analysis: Large pretrained models with web data
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Transformer architectures serve as the backbone for most modern Large Language Models, therefore their pretraining stability and convergence speed are of central concern. Motivated by the logical dependency of sequentially stacked layers, we propose Progressive Residual Warmup (ProRes) for language model pretraining. ProRes implements an"early layer learns first"philosophy by multiplying each layer's residual with a scalar that gradually warms up from 0 to 1, with deeper layers taking longer warmup steps. In this way, deeper layers wait for early layers to settle into a more stable regime before contributing to learning. We demonstrate the effectiveness of ProRes through pretraining experiments across various model scales, as well as normalization and initialization schemes. Comprehensive analysis shows that ProRes not only stabilizes pretraining but also introduces a unique optimization trajectory, leading to faster convergence, stronger generalization and better downstream performance. Our code is available at https://github.com/dandingsky/ProRes.

Problem

• pretraining stability
• convergence speed
• language model
• Transformer architecture

Innovation

• Progressive Residual Warmup
• language model pretraining
• residual scaling
• layer-wise warmup
• training stability