Learning single index model with gradient descent: spectral initialization and precise asymptotics

2025-09-27

AI Summary

This paper investigates the exact asymptotic dynamics of a two-stage algorithm—spectral initialization followed by gradient descent—for single-index models. Under the high-dimensional proportional limit (where $n, d o infty$ and $d/n o delta$), we derive the first dynamic mean-field equations that rigorously characterize both the transient evolution and steady-state distribution of the algorithm’s iterates. Our analysis reveals time-translation invariance and exponential convergence, yielding a mean-field characterization of the limiting fixed point. Methodologically, we depart from conventional fixed-point analysis by providing a full trajectory-level asymptotic description. Theoretically, we predict convergence rates and error evolution, and empirically validate our results via regularized Wirtinger flow on phase retrieval tasks. This work establishes a rigorous theoretical foundation for interpreting and designing nonconvex optimization algorithms in high dimensions.

Technology Category

• Search and Optimization: Non-convex Optimization
• Machine Learning: Learning with Manifolds
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Non-convex optimization plays a central role in many statistics and machine learning problems. Despite the landscape irregularities for general non-convex functions, some recent work showed that for many learning problems with random data and large enough sample size, there exists a region around the true signal with benign landscape. Motivated by this observation, a widely used strategy is a two-stage algorithm, where we first apply a spectral initialization to plunge into the region, and then run gradient descent for further refinement. While this two-stage algorithm has been extensively analyzed for many non-convex problems, the precise distributional property of both its transient and long-time behavior remains to be understood. In this work, we study this two-stage algorithm in the context of single index models under the proportional asymptotics regime. We derive a set of dynamical mean field equations, which describe the precise behavior of the trajectory of spectral initialized gradient descent in the large system limit. We further show that when the spectral initialization successfully lands in a region of benign landscape, the above equation system is asymptotically time translation invariant and exponential converging, and thus admits a set of long-time fixed points that represents the mean field characterization of the limiting point of the gradient descent dynamic. As a proof of concept, we demonstrate our general theory in the example of regularized Wirtinger flow for phase retrieval.

Problem

• Analyzing spectral initialization and gradient descent for single index models
• Deriving dynamical mean field equations for gradient descent trajectories
• Characterizing long-time behavior of two-stage non-convex optimization algorithms

Innovation

• Spectral initialization for gradient descent region entry
• Dynamical mean field equations for trajectory analysis
• Time translation invariance for exponential convergence characterization