Geometry of Learning -- L2 Phase Transitions in Deep and Shallow Neural Networks

2025-05-10

AI Summary

This work investigates phase transition phenomena in neural networks under L₂ regularization: beyond a critical regularization strength, models undergo a transition from overparameterized to underparameterized regimes—exhibiting first-order transitions in single-hidden-layer networks and second-order transitions in multi-hidden-layer networks. Methodologically, we integrate differential geometry (Ricci curvature analysis of the loss landscape), statistical physics (phase transition modeling), and deep learning theory, extending the framework to non-L₂ settings (e.g., VAE on MNIST). Our key contribution is the construction of a unified geometric framework that rigorously proves the exact coincidence between the singularity point of the loss landscape’s Ricci curvature and the critical point of generalization accuracy phase transitions. Empirically, Ricci curvature changes accurately predict sharp generalization performance jumps. This provides an interpretable, geometry-based foundation for principled regularization design and joint architecture–data optimization.

Technology Category

• Machine Learning: Learning with Manifolds
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

When neural networks (NNs) are subject to L2 regularization, increasing the regularization strength beyond a certain threshold pushes the model into an under-parameterization regime. This transition manifests as a first-order phase transition in single-hidden-layer NNs and a second-order phase transition in NNs with two or more hidden layers. This paper establishes a unified framework for such transitions by integrating the Ricci curvature of the loss landscape with regularizer-driven deep learning. First, we show that a curvature change-point separates the model-accuracy regimes in the onset of learning and that it is identical to the critical point of the phase transition driven by regularization. Second, we show that for more complex data sets additional phase transitions exist between model accuracies, and that they are again identical to curvature change points in the error landscape. Third, by studying the MNIST data set using a Variational Autoencoder, we demonstrate that the curvature change points identify phase transitions in model accuracy outside the L2 setting. Our framework also offers practical insights for optimizing model performance across various architectures and datasets. By linking geometric features of the error landscape to observable phase transitions, our work paves the way for more informed regularization strategies and potentially new methods for probing the intrinsic structure of neural networks beyond the L2 context.

Problem

• Analyzing L2 regularization-induced phase transitions in neural networks
• Linking Ricci curvature changes to model accuracy transitions
• Extending phase transition analysis beyond L2 regularization contexts

Innovation

• Integrates Ricci curvature with L2 regularization
• Identifies phase transitions via curvature change-points
• Extends framework beyond L2 to MNIST VAE