Learning time-scales in two-layers neural networks

2023-02-28

Foundations of Computational Mathematics

AI Summary

This work addresses the phenomenon of timescale separation and intermittent learning in gradient-based training of two-layer neural networks under single-index data distributions. We systematically uncover the asynchronous evolution between weight updates and hidden-layer feature learning. Methodologically, we formulate a continuous-time gradient flow model and integrate mean-field analysis, dynamical systems theory, and asymptotic expansion techniques to formally characterize the multi-scale dynamics of neuron activation and parameter updates. We rigorously prove that feature learning occurs significantly faster than weight convergence. Empirical validation confirms that this temporal hierarchy critically governs generalization performance. Our contribution establishes a theoretical framework for “fast-slow variable separation” in deep learning dynamics, offering a novel paradigm for understanding implicit regularization and phase-wise learning behaviors.

Problem

• Understanding non-monotone risk decrease in neural networks
• Explaining long plateaus and rapid learning phases
• Analyzing gradient flow dynamics in two-layer networks

Innovation

• Analyzes gradient flow in two-layer networks
• Uses singularly perturbed dynamical systems
• Combines rigorous results with simulations