The Map Behind the Flow: Finite-Step Gradient Descent as a Dynamical System

2026-07-06

AI Summary

This work addresses the inability of classical gradient flow models to capture dynamic phenomena induced by finite-step gradient descent, such as edge-of-stability behavior, sharpness oscillations, and representation selection. By modeling fixed-step gradient descent as a discrete dynamical system, the authors introduce an analytically tractable reduced model that preserves depth, factorization, and width structure to uncover the essential mechanisms governing training dynamics. They interpret the edge of stability as the first bifurcation of the training map and identify the learning rate as a structural parameter governing attractor formation and representation selection. Through scalar reduction, large-depth scaling, and analysis via Ricker-type maps, they demonstrate how finite step sizes break the conservation laws and contraction balance inherent in gradient flows, driving parameters toward flatter and more balanced representations. Moreover, they prove that the optimal learning rate can exceed the first-order spectral edge.

Abstract

Many phenomena of deep learning are dynamical: they concern not only which minima exist, but how gradient descent reaches, avoids, or selects among them. Edge-of-stability behavior, sharpness oscillations, catapult phases, balancing, and movement toward flatter representations are effects of the training map itself, and are poorly captured by the small-step gradient-flow limit. This paper studies fixed-step gradient descent as a discrete dynamical system in a hierarchy of exactly solvable models retaining basic structures of deep learning: depth, factorization, width, data coupling, activation, and stochasticity. The starting point is the balanced scalar reduction of a deep linear chain, giving a quartic loss and a cubic gradient map whose post-edge behavior is explicit. Under the natural large-depth scaling, this dynamics converges to a universal Ricker-type map. The edge of stability is therefore not a breakdown of optimization, but the first bifurcation of the training map. Embedding the scalar dynamics back into factored models turns these regimes into learning phenomena. Finite steps break conservation laws of gradient flow and contract factorization imbalance; residual oscillations move parameters toward flatter, more balanced representations. Wider linear networks produce a ladder of spectral edges, so the optimal learning rate can lie beyond the first edge. Data coupling, nonlinear activations, and stochastic targets preserve the same organizing principle: finite-step oscillations drive alignment, balancing, and representation selection. Thus the learning rate is not merely a numerical stability parameter. It is a structural parameter of the training dynamics, determining its attractors and shaping the representations gradient descent selects.

Problem

• gradient descent
• dynamical system
• edge of stability
• finite-step dynamics
• representation learning

Innovation

• discrete dynamical system
• edge of stability
• finite-step gradient descent
• Ricker map
• representation selection