From SGD to Spectra: A Theory of Neural Network Weight Dynamics

2025-07-16

AI Summary

Understanding the theoretical mechanisms governing the dynamic evolution of singular value spectra of weight matrices during deep neural network training—particularly the mapping between microscopic SGD dynamics and macroscopic spectral structure—remains an open challenge. Method: We propose a continuous-time modeling framework based on matrix-valued stochastic differential equations (SDEs), enabling rigorous analysis of weight matrix dynamics. Contribution/Results: We theoretically derive, for the first time, that the squared singular values evolve according to Dyson Brownian motion with repulsion, and we rigorously establish their stationary distribution as a Gamma-type distribution with power-law tails—explaining the empirically ubiquitous “bulk+tail” spectral structure. Integrating random matrix theory and spectral analysis, our predictions quantitatively match observed spectral evolution across MLPs and Transformers. This work provides the first analytically tractable and empirically verifiable spectral evolution theory for deep learning training dynamics.

Technology Category

• Machine Learning: Deep Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Models for Web evolution
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience

Abstract

Deep neural networks have revolutionized machine learning, yet their training dynamics remain theoretically unclear-we develop a continuous-time, matrix-valued stochastic differential equation (SDE) framework that rigorously connects the microscopic dynamics of SGD to the macroscopic evolution of singular-value spectra in weight matrices. We derive exact SDEs showing that squared singular values follow Dyson Brownian motion with eigenvalue repulsion, and characterize stationary distributions as gamma-type densities with power-law tails, providing the first theoretical explanation for the empirically observed 'bulk+tail' spectral structure in trained networks. Through controlled experiments on transformer and MLP architectures, we validate our theoretical predictions and demonstrate quantitative agreement between SDE-based forecasts and observed spectral evolution, providing a rigorous foundation for understanding why deep learning works.

Problem

• Modeling neural network weight dynamics via continuous-time SDEs
• Explaining bulk+tail spectral structure in trained networks
• Validating theoretical predictions with transformer and MLP experiments

Innovation

• Continuous-time matrix-valued SDE framework
• Dyson Brownian motion for singular values
• Gamma-type densities explain spectral structure