Gradient Flow Dynamics and Implicit Bias of Diagonal Linear Networks under Infinitesimal Initialization

2026-07-14

AI Summary

This work investigates the gradient flow dynamics and implicit bias of diagonal linear networks under infinitesimal initialization in regression tasks. By constructing an equivalent algorithm that characterizes the training trajectories of both deep and generalized two-layer diagonal linear networks, the study introduces a Structurally Invariant Manifold (SIM) to uncover the geometric mechanism underlying their learning dynamics. Theoretical analysis demonstrates that both network architectures converge to a modified ℓ₁-norm minimizer, thereby extending existing implicit bias theory to a broader class of diagonal linear structures and clarifying the optimization-driven nature of their sparsity-inducing behavior.

Abstract

We study the gradient flow dynamics of diagonal linear networks for regression tasks under infinitesimal initialization. Extending Theorem 1 from Pesme & Flammarion (2023), we generalize the analysis to both deep diagonal linear networks and a broader class of two-layer diagonal linear networks (as defined in Definition 4.1). Specifically, we demonstrate that the training trajectories of these models can be equivalently characterized by the proposed Algorithm 1. We further prove that this algorithm converges to the solution of a modified $ \mathcal{l}_1 $ norm minimization problem. As a result, we establish that the implicit bias of both network architectures corresponds to a modified $ \mathcal{l}_1 $ norm in the regime of infinitesimal initialization. Additionally, we provide insights into the underlying mechanisms governing these dynamics by identifying the Structural Invariant Manifold (SIM) (Zhao et al., 2026) as the key geometric structure that shapes the learning process.

Problem

• gradient flow
• implicit bias
• diagonal linear networks
• infinitesimal initialization
• regression

Innovation

• diagonal linear networks
• gradient flow dynamics
• implicit bias
• infinitesimal initialization
• Structural Invariant Manifold