Broken Ergodicity and the Violation of the Fluctuation-Dissipation Theorem Lead to Generalization Beyond Overfitting in Machine Learning

2026-07-05

AI Summary

This work investigates the mechanism underlying the surprising generalization ability of neural networks even when their parameter count exceeds the number of training samples, with a focus on the counterintuitive double descent phenomenon—where generalization error first increases and then decreases near a critical interpolation threshold. By modeling the training dynamics as a phase transition in a random field theory through dynamical mean-field theory, the study reveals for the first time that this behavior stems from ergodicity breaking and the breakdown of the fluctuation–dissipation theorem. Drawing an analogy to the London model of superconducting phase transitions, the authors introduce a notion of “generalization rigidity” and employ statistical physics techniques to compute the critical exponents and scaling functions governing the double descent transition, thereby providing a physical explanation for enhanced generalization in the over-parameterized regime.

Abstract

The remarkable ability of modern neural networks to generalize improves with increasing network capacity, even when the number of model parameters or effective degrees of freedom exceeds the number of training data points. This phenomenon is all the more surprising given that generalization error diverges when the number of model parameters approaches a critical value from below. Here we use dynamical mean field theory to show that this so-called"double descent"behavior is the outcome of a phase transition in the stochastic field theory describing the training process. We calculate the critical exponents and scaling function of the double descent phase transition, and show that it is marked by a breakdown of the fluctuation-dissipation theorem associated with broken ergodicity. The corresponding response function has the same functional form as the simple London model of the superconducting transition, with the rigidity of the wave function corresponding to the neural network's ability to generalize accurately.

Problem

• double descent
• generalization
• overfitting
• fluctuation-dissipation theorem
• broken ergodicity

Innovation

• double descent
• broken ergodicity
• fluctuation-dissipation theorem
• dynamical mean field theory
• phase transition