Manifold constrained steepest descent

2026-01-29

AI Summary

This work addresses the high computational cost of existing Linear Minimization Oracle (LMO)-based methods for manifold-constrained optimization, which typically rely on nested loops. To overcome this limitation, the authors propose MCSD, a single-loop framework that selects the steepest descent direction by applying a norm-constrained LMO to the Riemannian gradient and ensures feasibility via manifold projection, thereby eliminating inner iterations. Furthermore, they develop the SPEL algorithm tailored to the Stiefel manifold, leveraging the fast matrix sign function for efficient and scalable computation. By integrating stochastic momentum and spectral-norm-specific techniques, MCSD demonstrates superior stability and competitive performance over standard Riemannian methods and existing LMO approaches in tasks such as principal component analysis, orthogonal-constrained CNNs, and manifold-constrained adapter tuning for large language models.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Norm-constrained linear minimization oracle (LMO)-based optimizers such as spectral gradient descent and Muon are attractive in large-scale learning, but extending them to manifold-constrained problems is nontrivial and often leads to nested-loop schemes that solve tangent-space subproblems iteratively. We propose \emph{Manifold Constrained Steepest Descent} (MCSD), a single-loop framework for optimization over manifolds that selects a norm-induced steepest-descent direction via an LMO applied to the Riemannian gradient, and then returns to the manifold via projection. Under standard smoothness assumptions, we establish convergence guarantees for MCSD and a stochastic momentum variant. We further introduce \emph{SPEL}, the spectral-norm specialization of MCSD on the Stiefel manifold, which admits scalable implementations via fast matrix sign computations. Experiments on PCA, orthogonality-constrained CNNs, and manifold-constrained LLM adapter tuning demonstrate improved stability and competitive performance relative to standard Riemannian baselines and existing manifold-aware LMO methods.

Problem

• manifold-constrained optimization
• norm-constrained LMO
• steepest descent
• Riemannian optimization
• Stiefel manifold

Innovation

• Manifold optimization
• Linear Minimization Oracle (LMO)
• Steepest descent
• Stiefel manifold
• Single-loop algorithm