Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamics

2026-06-30

AI Summary

This work addresses the slow convergence often caused by improper rank selection in low-rank matrix optimization. To overcome this limitation, we propose a Direction-Magnitude Decomposition (DMD) framework that decouples direction and magnitude variables to enhance optimization efficiency, particularly in settings where the target rank is unknown. We introduce two novel variants—over-parameterized DMD and recursive DMD—that exploit the saddle-to-saddle dynamics observed along optimization trajectories, thereby significantly accelerating convergence. Theoretically, we establish that DMD achieves exponential acceleration over standard gradient descent. Empirical evaluations on matrix factorization, sensing, and completion tasks consistently demonstrate the superior efficiency of the proposed approach.

Abstract

Low-rank matrix optimization is often carried out via the Burer-Monteiro (BM) formulation, but choosing the factorization rank $r$ is delicate and can substantially slow optimization. We propose a unified framework, termed direction-magnitude decomposition (DMD), that decomposes the optimization variable to improve optimization efficiency even when the target rank is unknown. We develop two DMD-based approaches and establish their theoretical advantages on the canonical problem of matrix factorization. The first, overparameterized DMD, uses a rank $r$ larger than necessary and enjoys faster convergence as $r$ increases. The second, recursive DMD, is motivated by the incremental eigenpair learning, or saddle-to-saddle, behavior of overparameterized DMD. It achieves lower memory and computational costs, complementing overparameterized DMD. Both approaches are exponentially faster than gradient descent applied to the BM formulation. Numerical experiments on matrix factorization, sensing, and completion corroborate our theoretical findings and demonstrate the practical effectiveness of DMD.

Problem

• low-rank matrix optimization
• Burer-Monteiro formulation
• factorization rank selection
• optimization efficiency

Innovation

• Direction-Magnitude Decomposition
• Low-rank Matrix Optimization
• Overparameterization
• Saddle-to-saddle Dynamics
• Burer-Monteiro Formulation