Singular perturbations and hierarchical learning in two-layer neural networks

2026-07-12

AI Summary

This study investigates the hierarchical learning mechanism of infinitely wide two-layer neural networks when trained on misspecified single-index models in high dimensions. By introducing a perturbation parameter to modulate the disparity in training speeds between the two layers, and combining quantitative approximation theory for singularly perturbed flows near an integral-constraint manifold with dynamic analysis of the empirical weight measure, the work rigorously establishes—for the first time—that constant and linear components can be precisely recovered within the prediction-relevant timescale. It further reveals their essential role in enabling subsequent learning of quadratic components. The analysis explicitly identifies the thresholds and timescales governing this hierarchy and demonstrates that, during quadratic component learning, only a sparse subset of neurons exhibits significant growth, while the remainder reorganize to preserve the already-learned structure.

Abstract

We study the population gradient flow of an infinitely wide two-layer neural network learning a misspecified single-index model in high dimension. The two layers are optimized jointly, with a perturbative parameter tuning the relative training speed between the first and second layer. This setting was considered by Berthier, Montanari and Zhou in \cite{berthier2024learning}, who conjectured a hierarchical learning scenario with explicit timescales as the second layer is trained faster than the first. In this paper, we prove that the constant and linear components of the hidden link function are indeed recovered within the predicted timescales, at sharp explicit thresholds. We then analyze the onset of learning of the quadratic component and show that the components learned at earlier stages continue to influence the dynamics in an essential way. Our proof is based on quantitative approximation results for singularly perturbed flows evolving near a manifold defined by integral constraints. At a phenomenological level, we also show that the empirical measure of the weights displays singular behaviour when reaching the quadratic component of the hidden link, with a small fraction of neurons growing significantly while the remaining ones rearrange to preserve the components already learned.

Problem

• singular perturbations
• hierarchical learning
• two-layer neural networks
• single-index model
• population gradient flow

Innovation

• singular perturbations
• hierarchical learning
• two-layer neural networks
• gradient flow
• population dynamics