Muon$^p$: Muon with Fractional Spectral Powers

2026-06-11

AI Summary

This work addresses a critical limitation of the traditional Muon optimizer, which discards spectral information essential for adaptive optimization by fully flattening the singular spectrum of gradients. To remedy this, we propose Muon$^p$, a novel optimizer that introduces fractional spectral power updates of the form $US^pV^\top$ with $p \in (0,1)$, thereby interpolating between standard gradient descent and Muon to balance spectral structure preservation with optimization efficiency. We are the first to incorporate fractional spectral powers into optimizer design, proving that such operations cannot be computed via univariate polynomial iterations and instead devising a low-order bivariate recurrence approximation relying solely on matrix multiplications. Experiments demonstrate that Muon$^p$ significantly improves validation perplexity and downstream task performance in billion-scale model fine-tuning, while also revealing its operational boundaries from a spectral geometry perspective.

Abstract

Muon is an increasingly widely used optimizer that replaces a gradient $G=USV^\top$ with its polar factor $UV^\top$, thereby flattening the singular spectrum. However, full flattening discards singular-value information that may matter for adaptation. We introduce Muon$^p$, a Muon-style optimizer that instead uses fractional spectral-power updates $US^pV^\top$ for rational $p\in(0,1)$, interpolating between Muon and gradient descent. To make it practical, we prove that fractional spectral powers cannot be computed by any fixed univariate polynomial iteration, and furthermore derive low-degree odd bivariate recurrences that approximate $US^pV^\top$ using only matrix multiplications, preserving Muon's matrix-multiplication-only structure and compute complexity. We show that Muon$^p$ maximizes the linear improvement in loss under the Schatten $q$-norm for $q=1+\frac{1}{p}$. Empirically, Muon$^p$ is especially effective for finetuning: on billion-scale models, Muon$^p$ improves validation perplexity and downstream task performance. We further analyze when Muon$^p$ is less suitable, through the lens of spectral geometry. Our results reveal important insights on when preserving the singular spectrum can bring significant gains, and introduce a principled way to achieve them.

Problem

• optimization
• singular spectrum
• fractional spectral powers
• gradient descent
• matrix factorization

Innovation

• fractional spectral power
• matrix optimization
• Schatten norm
• bivariate recurrence
• singular spectrum preservation