An Equivariance Toolbox for Learning Dynamics

2025-12-24

AI Summary

Symmetries (equivariances) of neural network parameter transformations are crucial for understanding training dynamics and implicit bias, yet existing theories are largely restricted to specific settings and focus primarily on first-order conservation laws, lacking a systematic characterization of second-order curvature structure. Method: We develop a general equivariance toolkit, extending Noether-type analysis for the first time to discrete transformations, generalized equivariances, and Hessian-level constraints—unifying conservation laws and curvature geometry. Our approach integrates Lie group/Lie algebra modeling, continuous and discrete symmetry analysis, Hessian spectral geometry, and dynamical constraint derivation. Contribution/Results: The framework reproduces several classical conservation results and newly derives gradient-Hessian alignment principles, criteria distinguishing flat versus sharp optimization directions, and quantitative explanations for empirically observed curvature anisotropy during training—establishing a systematic link between transformation structure and optimization geometry.

Technology Category

• Machine Learning: Learning with Manifolds
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Many theoretical results in deep learning can be traced to symmetry or equivariance of neural networks under parameter transformations. However, existing analyses are typically problem-specific and focus on first-order consequences such as conservation laws, while the implications for second-order structure remain less understood. We develop a general equivariance toolbox that yields coupled first- and second-order constraints on learning dynamics. The framework extends classical Noether-type analyses in three directions: from gradient constraints to Hessian constraints, from symmetry to general equivariance, and from continuous to discrete transformations. At the first order, our framework unifies conservation laws and implicit-bias relations as special cases of a single identity. At the second order, it provides structural predictions about curvature: which directions are flat or sharp, how the gradient aligns with Hessian eigenspaces, and how the loss landscape geometry reflects the underlying transformation structure. We illustrate the framework through several applications, recovering known results while also deriving new characterizations that connect transformation structure to modern empirical observations about optimization geometry.

Problem

• Extends symmetry analysis to second-order learning dynamics constraints.
• Unifies conservation laws and implicit bias through equivariance framework.
• Connects transformation structure to loss landscape geometry predictions.

Innovation

• Extends Noether analysis to Hessian constraints and equivariance
• Unifies conservation laws and implicit bias via single identity
• Predicts curvature structure linking transformations to loss geometry