Percolation Dynamics in Optimization : Variance Cascades and Discrete Scale Invariance

2026-09-02

AI Summary

研究通过将随机梯度流建模为渗流过程，解释了SGD如何引导深度神经网络向更简单的子网络演化，并探讨了该机制在Adam和AdamW中的适用性。

Abstract

We study the dynamics of Stochastic Gradient Descent (SGD), which is known to steer deep neural networks toward invariant sets that correspond to simpler subnetworks. How this steering unfolds over time remains poorly understood. We answer this by modeling the stochastic gradient flow (SGF) as a percolation process, in which architectural symmetries force subnetworks to merge in discrete simultaneous blocks rather than one at a time. These structural transitions register as variance spikes in a macroscopic order parameter, echoing physical phase transitions. We further show this trapping mechanism and its associated scaling cascade extend to Adam and AdamW under an explicit heavy-tailed noise model.

Problem

• Stochastic Gradient Descent
• invariant sets
• percolation process
• variance spikes

Innovation

• percolation process
• discrete scale invariance
• variance spikes
• stochastic gradient flow (SGF)
• heavy-tailed noise model