Learning Rate Transfer in Normalized Transformers

2026-04-29

AI Summary

Although the Normalized Transformer (nGPT) offers training efficiency, its learning rate does not transfer well across variations in model width, depth, and context length. This work addresses this limitation by integrating alignment exponent theory with large-scale numerical experiments to refine the μP hyperparameter transfer methodology, yielding a novel parameterization scheme termed νGPT. The proposed νGPT enables, for the first time in nGPT architectures, successful learning rate transfer across diverse model widths, depths, and token horizons. It consistently maintains stable performance across a wide range of model scales and context lengths, substantially enhancing hyperparameter reusability, training robustness, and scalability.

Abstract

The Normalized Transformer, or nGPT (arXiv:2410.01131) achieves impressive training speedups and does not require weight decay or learning rate warmup. However, despite having hyperparameters that explicitly scale with model size, we observe that nGPT does not exhibit learning rate transfer across model dimension and token horizon. To rectify this, we combine numerical experiments with a principled use of alignment exponents (arXiv:2407.05872) to revisit and modify the $μ$P approach to hyperparameter transfer (arXiv:2011.14522). The result is a novel nGPT parameterization we call $ν$GPT. Through extensive empirical validation, we find $ν$GPT exhibits learning rate transfer across width, depth, and token horizon.

Problem

• learning rate transfer
• Normalized Transformer
• model scaling
• hyperparameter transfer
• token horizon

Innovation

• learning rate transfer
• Normalized Transformer
• alignment exponents
• hyperparameter scaling
• νGPT