Intrinsic Muon: Spectral Optimization on Riemannian Matrix Manifolds

2026-05-09

AI Summary

Existing Muon optimizers struggle to effectively handle manifold-constrained parameters—such as those with low-rank, orthogonality, or symmetric positive-definite (SPD) structures—because their use of the Euclidean norm in tangent spaces breaks symmetry and precludes closed-form solutions. This work proposes an intrinsic Muon (iMuon) framework that, for the first time, lifts unitarily invariant norms to tangent spaces via the Riemannian metric, yielding a symmetry-preserving, norm-constrained linear maximization oracle. This enables closed-form updates on diverse matrix manifolds, including fixed-rank, SPD, Stiefel, and Grassmann manifolds. The method requires neither factor conditioning nor runtime rescaling and demonstrates strong empirical performance in LoRA fine-tuning of large language models, image classification, and subspace learning. Its convergence rate depends only on intrinsic manifold dimensions, such as rank.

Abstract

Muon and related norm-constrained matrix optimizers have become central to large-scale learning problems. They are formulated as a linear maximization oracle (LMO) over an ambient matrix-norm ball in unconstrained Euclidean space. However, these do not generalize cleanly to manifold-valued parameters such as low-rank factorizations, orthogonality constraints, or symmetric positive definite (SPD) matrices. Naively restricting the Muon LMO to the tangent space (i) breaks quotient symmetries and (ii) couples the tangent-space constraint with an ambient norm bound, thereby obstructing closed-form solutions on various manifolds of interest. We resolve both issues with a single observation: every Riemannian metric canonically lifts a unitarily invariant Euclidean norm to an intrinsic norm on each tangent space, and the resulting intrinsic norm constrained LMO is symmetry preserving. Building on this, we introduce intrinsic Muon (iMuon), a unified framework that yields closed-form updates on the fixed-rank, SPD, Stiefel, and Grassmann manifolds for any unitarily invariant norm, including the spectral, Frobenius, and nuclear norms. We establish convergence guarantees for both deterministic and stochastic iMuon with rate constants that depend only on the manifold dimension. Notably, on the fixed-rank manifold this constant depends only on the rank, making the rate independent of factor conditioning and removing the runtime factor-rescaling required by prior work. Experiments on LoRA finetuning of LLMs, image classification, and subspace learning illustrate the efficacy of the proposed approach.

Problem

• Riemannian manifold
• matrix optimization
• norm-constrained optimization
• symmetry preservation
• intrinsic geometry

Innovation

• intrinsic Muon
• Riemannian manifold optimization
• unitarily invariant norm
• closed-form update
• symmetry preservation