Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs
This work addresses the optimization of learning rate (LR) decay scheduling in large language model (LLM) training. We systematically evaluate diverse LR schedulers and find that linear decay to zero (D2Z) consistently outperforms mainstream alternatives—including cosine decay to 10%—under computation-optimal token-per-parameter (TPP) scaling. Theoretically, we provide the first analysis showing that D2Z achieves an optimal trade-off: accelerating escape from suboptimal initial parameter regions early in training while effectively suppressing gradient noise in later stages; we further reinterpret AdamW’s exponential moving average of weight updates through this lens. Empirically, under TPP-controlled scaling across models from 610M to 7B parameters, D2Z enables the 610M model to achieve lower loss at just 80 TPP—reducing compute cost by 60% versus a 10× decay baseline requiring 200 TPP—and delivers substantial training efficiency gains for larger models including Llama2-7B.