Insights on Muon from Simple Quadratics

2026-02-12

AI Summary

Existing theoretical analyses struggle to explain the superior performance of the Muon optimizer on strongly convex functions, as they rely on local quadratic approximations and worst-case bounds that overlook the impact of polar factorization errors and the structure of the objective function on finite-step dynamics. This work analyzes the discrete-time dynamics of Muon on strongly convex quadratics and reveals that polar factorization error is not merely a compromise in precision but can actively enhance optimization reachability and finite-step performance. Furthermore, it demonstrates that the influence of the objective function’s structure extends beyond the classical condition number, necessitating its incorporation into a more comprehensive theoretical framework. By transcending the limitations of current Muon theory, this study lays the groundwork for developing more accurate and insightful optimizer analyses.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements

Abstract

Muon updates weight matrices along (approximate) polar factors of the gradients and has shown strong empirical performance in large-scale training. Existing attempts at explaining its performance largely focus on single-step comparisons (on quadratic proxies) and worst-case guarantees that treat the inexactness of the polar-factor as a nuisance ``to be argued away''. We show that already on simple strongly convex functions such as $L(W)=\frac12\|W\|_{\text{F}}^2$, these perspectives are insufficient, suggesting that understanding Muon requires going beyond local proxies and pessimistic worst-case bounds. Instead, our analysis exposes two observations that already affect behavior on simple quadratics and are not well captured by prevailing abstractions: (i) approximation error in the polar step can qualitatively alter discrete-time dynamics and improve reachability and finite-time performance -- an effect practitioners exploit to tune Muon, but that existing theory largely treats as a pure accuracy compromise; and (ii) structural properties of the objective affect finite-budget constants beyond the prevailing conditioning-based explanations. Thus, any general theory covering these cases must either incorporate these ingredients explicitly or explain why they are irrelevant in the regimes of interest.

Problem

• Muon
• polar decomposition
• optimization dynamics
• approximation error
• quadratic functions

Innovation

• Muon optimizer
• polar decomposition
• approximation error
• discrete-time dynamics
• structural properties