On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness

2026-03-27

AI Summary

This work investigates the geometric properties of the optimization landscape for ℓ²-regularized deep matrix factorization under squared error loss, with a focus on the uniqueness and sharpness of end-to-end minimizers. By integrating tools from measure theory, Hessian spectral analysis, and Frobenius norm characterizations, the authors prove that, outside a set of measure zero, every target matrix admits a unique global minimizer, at which the norms of all layer parameters and the Hessian spectrum remain constant. Furthermore, they establish a global lower bound on the trace of the Hessian at minimizers and precisely identify the critical regularization strength above which the solution collapses to zero.

Abstract

Weight decay is ubiquitous in training deep neural network architectures. Its empirical success is often attributed to capacity control; nonetheless, our theoretical understanding of its effect on the loss landscape and the set of minimizers remains limited. In this paper, we show that $\ell^2$-regularized deep matrix factorization/deep linear network training problems with squared-error loss admit a unique end-to-end minimizer for all target matrices subject to factorization, except for a set of Lebesgue measure zero formed by the depth and the regularization parameter. This observation reveals fundamental properties of the loss landscape of regularized deep matrix factorization problems: the Hessian spectrum is constant across all minimizers of the regularized deep scalar factorization problem with squared-error loss. Moreover, we show that, in regularized deep matrix factorization problems with squared-error loss, if the target matrix does not belong to the Lebesgue measure-zero set, then the Frobenius norm of each layer is constant across all minimizers. This, in turn, yields a global lower bound on the trace of the Hessian evaluated at any minimizer of the regularized deep matrix factorization problem. Furthermore, we establish a critical threshold for the regularization parameter above which the unique end-to-end minimizer collapses to zero.

Problem

• loss landscape
• deep matrix factorization
• regularization
• minimizers
• Hessian spectrum

Innovation

• deep matrix factorization
• loss landscape geometry
• ℓ² regularization
• unique minimizer
• Hessian spectrum