The Complexity Dynamics of Grokking

2024-12-13

arXiv.org

AI Summary

This work investigates the complexity phase transition underlying the shift from overfitting to generalization in neural network training, focusing on the intrinsic compression dynamics that drive generalization emergence (“insight”). Method: We propose a computable intrinsic complexity measure grounded in Kolmogorov complexity and establish, for the first time, a causal link between complexity evolution and generalization transitions: intrinsic complexity peaks mid-training and then monotonically decreases, with this decline strictly coinciding with the onset of generalization emergence. Leveraging this insight, we design spectral entropy regularization—a novel regularization technique—and theoretically derive the first provable generalization bound based on intrinsic complexity evolution. Results: Our method achieves superior model compression and enhanced generalization on CIFAR-10/100. The framework unifies rate-distortion theory, minimum description length, and deep learning dynamics, offering a new paradigm for understanding deep learning’s inductive bias.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We investigate the phenomenon of generalization through the lens of compression. In particular, we study the complexity dynamics of neural networks to explain grokking, where networks suddenly transition from memorizing to generalizing solutions long after over-fitting the training data. To this end we introduce a new measure of intrinsic complexity for neural networks based on the theory of Kolmogorov complexity. Tracking this metric throughout network training, we find a consistent pattern in training dynamics, consisting of a rise and fall in complexity. We demonstrate that this corresponds to memorization followed by generalization. Based on insights from rate--distortion theory and the minimum description length principle, we lay out a principled approach to lossy compression of neural networks, and connect our complexity measure to explicit generalization bounds. Based on a careful analysis of information capacity in neural networks, we propose a new regularization method which encourages networks towards low-rank representations by penalizing their spectral entropy, and find that our regularizer outperforms baselines in total compression of the dataset.

Problem

• Studying complexity phase transition in neural networks
• Characterizing grokking phenomenon from memorization to generalization
• Measuring network complexity via compression and regularization

Innovation

• Rate-distortion theory measures network complexity
• Spectral entropy regularization reduces intrinsic dimension
• Lossy compression framework tracks complexity dynamics