The Role of Symmetry in Optimizing Overparameterized Networks

2026-04-27

AI Summary

This work investigates how over-parameterization improves neural network optimization by introducing symmetries in weight space. We establish, for the first time, a formal connection between over-parameterization and symmetry, proposing a unified geometric framework that elucidates how increasing network width structurally reshapes the loss landscape. Theoretical analysis, supported by teacher-student network experiments, demonstrates that the symmetries induced by widening act equivalently to a diagonal preconditioner on the Hessian, yielding better-conditioned minima. Moreover, this effect concentrates probability mass around global minima near typical initializations, enhancing their accessibility. Empirical results corroborate these predictions: as width increases, the Hessian trace decreases, its condition number improves, and convergence accelerates, collectively validating the proposed theory.

Abstract

Overparameterization is central to the success of deep learning, yet the mechanisms by which it improves optimization remain incompletely understood. We analyze weight-space symmetries in neural networks and show that overparameterization introduces additional symmetries that benefit optimization in two distinct ways. First, we prove that these symmetries act as a form of diagonal preconditioning on the Hessian, enabling the existence of better-conditioned minima within each equivalence class of functionally identical solutions. Second, we show that overparameterization increases the probability mass of global minima near typical initializations, making these favorable solutions more reachable. Teacher-student network experiments validate our theoretical predictions: as width increases, the Hessian trace decreases, condition numbers improve, and convergence accelerates. Our analysis provides a unified framework for understanding overparameterization and width growth as a geometric transformation of the loss landscape.

Problem

• overparameterization
• symmetry
• optimization
• neural networks
• loss landscape

Innovation

• symmetry
• overparameterization
• Hessian preconditioning
• loss landscape geometry
• global minima reachability