Sharpness of Minima in Deep Matrix Factorization: Exact Expressions

2025-09-30

AI Summary

This work investigates the geometric structure near minima of the loss landscape in deep matrix factorization—i.e., training deep linear networks—with a focus on precisely characterizing “sharpness,” defined as the largest eigenvalue of the Hessian, to explain gradient-based optimization’s implicit bias in non-convex, over-parameterized regimes. Addressing an open problem posed by Mulayoff and Michaeli, we derive, for the first time, a closed-form expression for the maximum Hessian eigenvalue of the squared-error loss at any global minimum. Leveraging both theoretical analysis and numerical experiments, we demonstrate that this sharpness measure governs the escape dynamics of gradient descent in the vicinity of minima. Our results provide an exact theoretical characterization of sharpness and empirically validate its critical role in explaining training behavior—including convergence paths, generalization trends, and implicit regularization effects—thereby bridging theory and practice in deep linear learning.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eigenvalue of the Hessian of the loss, which measures the sharpness of the landscape. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in general overparameterized deep matrix factorization (i.e., deep linear neural network training) problems, resolving an open question posed by Mulayoff & Michaeli (2020). To complement our theory, we empirically investigate an escape phenomenon observed during gradient-based training near a minimum that crucially relies on our exact expression of the sharpness.

Problem

• Exact expression for Hessian maximum eigenvalue in deep matrix factorization
• Characterizing loss landscape geometry near minima in non-convex optimization
• Resolving open question about sharpness measure in overparameterized networks

Innovation

• Exact expression for Hessian maximum eigenvalue derived
• Sharpness characterization in deep matrix factorization achieved
• Overparameterized deep linear networks analyzed theoretically