A Link between Shock-wave Theory and Symmetry-reduced Stochastic Gradient Descent for Artificial Neural Networks

2026-06-16

AI Summary

This work addresses the challenge posed by parameter symmetries in deep neural network training, which induce redundancy and impede analytical understanding of training dynamics and phase transitions. For the first time, it integrates shock wave theory, differential geometry, and Lie group symmetry reduction into the analysis of deep learning dynamics. By performing symmetry reduction on quotient manifolds and applying local entropy-based coarse-graining, the study establishes a rigorous mathematical connection between stochastic gradient descent dynamics and viscous Hamilton–Jacobi or Burgers-type partial differential equations. The proposed observables defined on the quotient space effectively diagnose training phase transitions and demonstrate universality across diverse architectures—including multilayer perceptrons, convolutional networks, Transformers, and mean-field networks—thereby offering a principled framework for monitoring and controlling the training process.

Abstract

We develop a mathematically explicit link between shock-wave theory and the symmetry-quotiented learning dynamics of stochastic gradient descent, drawing on differential geometry, Lie group theory, and fluid mechanics. Specifically, after quotienting parameter symmetries and applying local-entropy coarse-graining, the effective dynamics satisfy a viscous Hamilton--Jacobi equation on the quotient manifold. Moreover, under the assumption that the raw parameter dynamics can be summarized by a gradient field on the quotiented space, the gradient of the coarse-grained loss function obeys a Burgers-type equation, and shock formation can be established rigorously. We apply our theory to multilayer perceptrons, convolutional neural networks, Transformers, and mean-field networks, and show that they obey the Hamilton--Jacobi or Burgers-type equations. We conjecture that this framework also yields practical diagnostics for deep learning. In architectures such as Transformers, raw parameter norms are often distorted by symmetry redundancy and may therefore be misleading, whereas symmetry-corrected quotient observables provide a principled basis for monitoring, forecasting, and controlling training-phase transitions.

Problem

• symmetry redundancy
• training-phase transitions
• parameter symmetries
• deep learning diagnostics
• quotient observables

Innovation

• shock-wave theory
• symmetry reduction
• stochastic gradient descent
• Hamilton–Jacobi equation
• Burgers equation