The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training

2026-05-25

AI Summary

This work uncovers the intrinsic mechanism behind the ubiquitous “rapid descent–slow optimization” two-phase dynamics observed during large language model pretraining. Through a spectral analysis lens, it introduces— for the first time—the phenomenon of Singular value distribution Stability (SoSD), revealing its strong temporal alignment with the slow optimization phase. Leveraging a simplified Transformer model, the study establishes theoretical bounds linking SoSD to weight norm growth and loss decay rates, thereby providing a unified spectral interpretation framework for optimization strategies such as WSD and Muon. The universality of SoSD is empirically validated across diverse architectures—including GPT-2 and LLaMA—and various training configurations, demonstrating its critical role in shaping pretraining efficiency.

Abstract

Large language model pre-training typically exhibits a two-phase trajectory: a fast initial loss drop followed by a prolonged slow improvement. We identify an underlying spectral phenomenon, Stability of Singular Distribution (SoSD), where the trace-normalized singular value spectrum stabilizes early, even as parameter matrices continue to evolve. We demonstrate that synchronization between SoSD and the slow-descent regime is widely observed across diverse architectures (GPT-2, LLaMA) and settings, including various schedules (Step-wise, WSD, Cosine Decay), weight decays, and optimizers (AdamW, Muon). By analyzing a simplified Transformer, we prove that growing weight norms inevitably precipitate an early SoSD threshold, after which the rate of loss decrease becomes theoretically bounded by the variation in the singular distribution. We further interpret strategies like WSD and Muon through their ability to modulate the SoSD scale, offering a spectral lens for understanding efficient pre-training dynamics.

Problem

• two-phase dynamics
• language model pre-training
• singular value spectrum
• spectral stability
• loss trajectory

Innovation

• Stability of Singular Distribution
• spectral analysis
• two-phase dynamics
• pre-training
• singular value spectrum