Replica Symmetry Breaking and Algorithmic Thresholds in Empirical Risk Minimization under Multi-Index Model

2026-06-26

AI Summary

Although high-dimensional non-convex empirical risk functions possess numerous local minima, gradient-based algorithms often converge to solutions near the global optimum; however, the precise characterization of the polynomial-time reachable region remains elusive. This work addresses this gap in the context of multi-index supervised learning models by integrating replica symmetry breaking theory with the Incremental Approximate Message Passing (IAMP) algorithm. Through high-dimensional asymptotic analysis within the empirical risk minimization framework, the study precisely characterizes the training error achievable by IAMP and establishes its quantitative relationship with the test error. The results delineate the performance boundary between computational feasibility and statistical optimality, demonstrating that IAMP achieves optimal performance among all polynomial-time algorithms.

Abstract

Modern machine learning models are trained by optimizing high-dimensional non-convex empirical risk functions. Such cost functions can have a multitude of local optima and yet, gradient-based optimization appears to converge to near-global optima. Within a simple supervised learning setting, we develop a precise picture of which parts of the empirical risk landscape are accessible by polynomial-time algorithms. We are given i.i.d. pairs $\{(\boldsymbol{x}_i,y_i):\; 1 \le i\le n\}$ with $\boldsymbol{x}_i\in \mathbb{R}^d$ standard Gaussian feature vectors, and $y_i\in\mathbb{R}$ response variables that depend on $\boldsymbol{x}_i$ through their projections on an unknown $k$-dimensional subspace. We use empirical risk minimization to learn a model that depends on an $m$-dimensional projection of the data (e.g., an $m$-neurons neural network). We propose an incremental approximate message passing (IAMP) algorithm and precisely characterize the training error it achieves, as well as the relation between test and training error, in the high dimensional asymptotics $n,d\to\infty$, with $n/d\toα\in (0, +\infty)$. Based on earlier work in related models, we expect that the performance achieved by our algorithm is optimal among polynomial-time algorithms.

Problem

• Empirical Risk Minimization
• Replica Symmetry Breaking
• Algorithmic Thresholds
• High-dimensional Asymptotics
• Multi-Index Model

Innovation

• Incremental Approximate Message Passing
• Replica Symmetry Breaking
• Empirical Risk Minimization
• Algorithmic Thresholds
• High-dimensional Asymptotics