Transformative or Conservative? Conservation laws for ResNets and Transformers

2025-06-06

AI Summary

This paper investigates conservation laws in the gradient flow training dynamics of modern deep networks—specifically convolutional ResNets and Transformers—and their robustness under discrete optimization (e.g., SGD). Methodologically, it establishes the first unified theoretical framework for conservation laws encompassing both architectures, integrating continuous gradient flow modeling, Lie group symmetry analysis, and modular decoupling techniques to rigorously characterize all conservation laws within individual attention layers and residual blocks. The key contributions are threefold: (1) a proof that residual connections introduce no new conservation laws—the global conserved quantities are fully determined by base components (e.g., linear layers, attention heads, normalization modules); (2) the introduction of “submodule-dependent conservation laws,” a novel paradigm enabling dimensionality reduction from global to local analysis; and (3) joint theoretical and empirical validation demonstrating the high robustness of these conservation laws under discrete optimizers such as SGD.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Large Vision Models
• Search and Optimization: Non-convex Optimization

Application Category

• Responsible Web: Consent frameworks and practices on the web
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

While conservation laws in gradient flow training dynamics are well understood for (mostly shallow) ReLU and linear networks, their study remains largely unexplored for more practical architectures. This paper bridges this gap by deriving and analyzing conservation laws for modern architectures, with a focus on convolutional ResNets and Transformer networks. For this, we first show that basic building blocks such as ReLU (or linear) shallow networks, with or without convolution, have easily expressed conservation laws, and no more than the known ones. In the case of a single attention layer, we also completely describe all conservation laws, and we show that residual blocks have the same conservation laws as the same block without a skip connection. We then introduce the notion of conservation laws that depend only on a subset of parameters (corresponding e.g. to a pair of consecutive layers, to a residual block, or to an attention layer). We demonstrate that the characterization of such laws can be reduced to the analysis of the corresponding building block in isolation. Finally, we examine how these newly discovered conservation principles, initially established in the continuous gradient flow regime, persist under discrete optimization dynamics, particularly in the context of Stochastic Gradient Descent (SGD).

Problem

• Study conservation laws in modern architectures like ResNets and Transformers
• Analyze conservation laws for building blocks like ReLU and attention layers
• Examine conservation principles under discrete optimization dynamics like SGD

Innovation

• Derives conservation laws for ResNets and Transformers
• Introduces subset-dependent conservation laws
• Examines conservation laws under discrete optimization