Conservation Laws for Modern Neural Architectures

2026-06-16

AI Summary

This work addresses the limited understanding of conservation laws governing gradient flow in modern neural networks, which hinders explanations of the implicit bias in over-parameterized models. We systematically extend conservation law theory to mainstream architectures, including feedforward networks with GELU, SiLU, or SwiGLU activations; multi-head attention mechanisms incorporating sinusoidal and rotary positional encodings; and mixture-of-experts models featuring diverse gating designs. By integrating gradient flow analysis, invariant theory from differential equations, and deep learning architecture modeling, we derive key conserved quantities for each architecture and empirically validate their existence and stability. Our findings uncover invariant structures within gradient dynamics, offering a novel perspective for understanding the inductive biases inherent in contemporary neural networks.

Abstract

Understanding gradient descent dynamics is key to explaining the success of over-parameterized models, where implicit bias manifests through conservation laws in gradient flow. While such laws are well understood for linear and ReLU networks, they remain largely unexplored for modern architectures. This work develops a unified framework to characterize conservation laws for contemporary models, including feedforward networks with GELU, SiLU, and SwiGLU activations, multihead attention with sinusoidal and rotary positional encodings, and Mixture-of-Experts architectures under diverse gating designs. Our theoretical findings are supported by experiments that validate the predicted invariants.

Problem

• conservation laws
• modern neural architectures
• gradient flow
• implicit bias
• over-parameterized models

Innovation

• conservation laws
• implicit bias
• modern neural architectures
• gradient flow
• Mixture-of-Experts