🤖 AI Summary
This work addresses the issue in the Muon optimizer where highly anisotropic row norms in tall-and-skinny matrices—such as those in MLP projection layers—lead to severely imbalanced neuron updates, causing some neurons to receive negligible or ineffective gradients. To resolve this, the authors propose the Aurora optimizer, which introduces a row-normalization mechanism while preserving the geometric structure of the momentum matrix’s polar factor. Aurora is the first method to achieve uniformly scaled row-wise updates without compromising this intrinsic geometry—a limitation inherent in prior approaches that trade geometric fidelity for update uniformity. By integrating spectral optimization techniques with polar factor constraints, Aurora outperforms Muon in pretraining and achieves state-of-the-art performance among spectral optimizers on the modded-nanoGPT speedrun benchmark, with gains that become increasingly pronounced as the MLP expansion factor grows.
📝 Abstract
We show that for tall matrix parameters, like projection matrices in the MLP layers, the Muon update can have row norms that are arbitrarily non-uniform. This can lead to a self-reinforcing feedback loop whereby neurons receive persistently small updates and eventually do not contribute meaningfully to network outputs. This problem is effectively mitigated by an additional row normalization step, but current methods do this in a way that moves the Muon update geometry away from the polar factor of the momentum matrix, which we find is undesirable. We propose Aurora, an optimizer that enforces row-uniformity of matrix parameter updates while respecting Muon's polar factor geometry. Aurora outperforms Muon in our pre-training experiments and, when combined with existing methods, achieves state-of-the-art performance among spectral optimizers on the optimizer track of the modded-nanoGPT speedrun. Additionally, we find that Aurora's empirical gains over Muon scale with the MLP expansion factor, suggesting that Aurora may allow for effective training of very wide MLP layers.