Spectral phase transitions and trainability in neural network learning dynamics

2026-06-26

AI Summary

This work uncovers the dynamical origin of low-dimensional spectral structure in weight matrices during neural network training and its connection to trainability. By modeling stochastic gradient descent (SGD) as the evolution of a random matrix ensemble, the study demonstrates that learning reshapes the spectral distribution and amplifies signal components, thereby inducing a Baik–Ben Arous–Péché (BBP) phase transition—a phenomenon newly introduced into neural network dynamics. Leveraging a linear teacher–student model combined with random matrix theory, the authors construct a trainability phase diagram governed by learning rate and initial weight variance, which is further extended to nonlinear and stochastic settings. Theoretical analysis yields precise spectral evolution laws, while numerical experiments on realistic architectures confirm the robust emergence of spectral alignment, establishing a unified spectral dynamical framework linking optimization hyperparameters to representation learning.

Abstract

The emergence of low-dimensional structures in the spectra of neural network weight matrices is a common empirical feature of trained models, but the dynamical origin of this phenomenon during learning remains an open problem. We formulate neural network training as the stochastic evolution of an initially random matrix ensemble, driven by stochastic gradient descent (SGD) updates that reshape the spectral bulk while amplifying signal strength. This induces a Baik-Ben Arous-Péché (BBP) transition during training, where isolated eigenvalues detach from the random bulk distribution, providing a dynamical framework for representation formation in high-dimensional learning dynamics. We demonstrate this in a solvable linear teacher-student model, where spectral evolution is analytically tractable and a phase diagram of trainability governed by the step size (or learning rate) and initial weight variance is obtained, and subsequently extend our formalism beyond the linear regime to nonlinear and stochastic settings. Numerical simulations in realistic settings support this picture, showing robust emergence of spectral alignment during training. Our results suggest that spectral analysis may provide a unified perspective of stochastic learning dynamics, linking trainability, optimisation hyperparameters, spectral phase transitions, and representation learning in neural networks.

Problem

• spectral phase transitions
• trainability
• neural network learning dynamics
• representation learning
• random matrix theory

Innovation

• spectral phase transition
• BBP transition
• trainability
• random matrix theory
• representation learning