Topological Exploration of High-Dimensional Empirical Risk Landscapes: general approach, and applications to phase retrieval

2026-02-19

AI Summary

This work addresses the challenge of characterizing the distribution of critical points in the empirical risk landscape and its impact on optimization dynamics under high-dimensional Gaussian single-index models. By leveraging the Kac–Rice formula within a proportional asymptotic regime, the authors reduce the original problem to a tractable finite-dimensional scalar variational problem. This approach yields the first complete topological phase diagram of the loss landscape for phase retrieval, along with a precise characterization of a BBP-type spectral instability transition in the Hessian along the signal direction. Integrating high-dimensional asymptotic analysis, Hessian spectral theory, and gradient flow simulations, the framework accurately predicts the distribution of critical points, their joint label statistics, and the dynamical behavior of local optimization algorithms, with theoretical predictions showing excellent agreement with finite-dimensional numerical experiments.

Technology Category

• Search and Optimization: Non-convex Optimization
• Reasoning under Uncertainty: Stochastic Optimization
• Machine Learning: Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

We consider the landscape of empirical risk minimization for high-dimensional Gaussian single-index models (generalized linear models). The objective is to recover an unknown signal $\boldsymbol{\theta}^\star \in \mathbb{R}^d$ (where $d \gg 1$) from a loss function $\hat{R}(\boldsymbol{\theta})$ that depends on pairs of labels $(\mathbf{x}_i \cdot \boldsymbol{\theta}, \mathbf{x}_i \cdot \boldsymbol{\theta}^\star)_{i=1}^n$, with $\mathbf{x}_i \sim \mathcal{N}(0, I_d)$, in the proportional asymptotic regime $n \asymp d$. Using the Kac-Rice formula, we analyze different complexities of the landscape -- defined as the expected number of critical points -- corresponding to various types of critical points, including local minima. We first show that some variational formulas previously established in the literature for these complexities can be drastically simplified, reducing to explicit variational problems over a finite number of scalar parameters that we can efficiently solve numerically. Our framework also provides detailed predictions for properties of the critical points, including the spectral properties of the Hessian and the joint distribution of labels. We apply our analysis to the real phase retrieval problem for which we derive complete topological phase diagrams of the loss landscape, characterizing notably BBP-type transitions where the Hessian at local minima (as predicted by the Kac-Rice formula) becomes unstable in the direction of the signal. We test the predictive power of our analysis to characterize gradient flow dynamics, finding excellent agreement with finite-size simulations of local optimization algorithms, and capturing fine-grained details such as the empirical distribution of labels. Overall, our results open new avenues for the asymptotic study of loss landscapes and topological trivialization phenomena in high-dimensional statistical models.

Problem

• empirical risk minimization
• high-dimensional statistics
• loss landscape
• phase retrieval
• critical points

Innovation

• Kac-Rice formula
• empirical risk landscape
• high-dimensional asymptotics
• phase retrieval
• topological phase transition