Local minima of the empirical risk in high dimension: General theorems and convex examples

2025-02-04

AI Summary

This work investigates the distribution and impact of local minima in high-dimensional empirical risk minimization, under isotropic Gaussian data and projection-type matrix parameterizations—including generalized linear models and two-layer neural networks—in the proportional asymptotic regime where $n,d o infty$ and $n asymp d$. Methodologically, it is the first to rigorously apply the Kac–Rice formula to characterize, for convex losses, the existence, location, and precise asymptotic spectrum of the Hessian at local minima of order $k geq 2$, thereby confirming a long-standing conjecture. Combining Gaussian process theory, random matrix analysis, and large deviation estimates, the paper derives a tight upper bound on the expected number of local minima and establishes sharp asymptotic expressions for both estimation and prediction errors, achieving exponential-level deviation control.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

We consider a general model for high-dimensional empirical risk minimization whereby the data $mathbf{x}_i$ are $d$-dimensional isotropic Gaussian vectors, the model is parametrized by $mathbf{Theta}inmathbb{R}^{d imes k}$, and the loss depends on the data via the projection $mathbf{Theta}^mathsf{T}mathbf{x}_i$. This setting covers as special cases classical statistics methods (e.g. multinomial regression and other generalized linear models), but also two-layer fully connected neural networks with $k$ hidden neurons. We use the Kac-Rice formula from Gaussian process theory to derive a bound on the expected number of local minima of this empirical risk, under the proportional asymptotics in which $n,d oinfty$, with $nasymp d$. Via Markov's inequality, this bound allows to determine the positions of these minimizers (with exponential deviation bounds) and hence derive sharp asymptotics on the estimation and prediction error. In this paper, we apply our characterization to convex losses, where high-dimensional asymptotics were not (in general) rigorously established for $kge 2$. We show that our approach is tight and allows to prove previously conjectured results. In addition, we characterize the spectrum of the Hessian at the minimizer. A companion paper applies our general result to non-convex examples.

Problem

• Analyzes local minima in high-dimensional empirical risk minimization.
• Derives bounds for local minima using Kac-Rice formula.
• Characterizes Hessian spectrum at minimizers for convex losses.

Innovation

• Kac-Rice formula analysis
• High-dimensional empirical risk
• Convex loss asymptotics