Mildly Overparameterized ReLU Networks on Orthogonal Data: Incremental Learning and Implicit Bias

2026-05-26

AI Summary

This work investigates the convergence and generalization mechanisms of mildly over-parameterized two-layer ReLU networks—where the width satisfies \( m \gtrsim \log n \)—trained via gradient flow on orthogonal data. Under small initialization, the study provides the first rigorous proof that the network learns incrementally by sequentially activating neurons and converges, with high probability, to an interpolating solution. This solution exhibits an ℓ² norm on the order of \( \sqrt{n} \), matching that of the minimum-norm interpolator, thereby revealing an implicit bias induced by optimization. By integrating gradient flow dynamics, saddle-to-saddle transition modeling, and asymptotic analysis in the small-initialization regime, the analysis elucidates the intrinsic mechanisms enabling both efficient learning and strong generalization precisely at the threshold of mild over-parameterization.

Abstract

The successful training of neural networks hinges on the use of first order optimization methods, yet the theoretical characterization of these methods remains incomplete. This is especially true in settings with mild overparameterization. In this work, we study the gradient flow dynamics of two-layer ReLU networks from small initialization with orthogonal training data. We prove the limiting flow converges to a saddle-to-saddle jump process as the initialization scale tends to zero, revealing an incremental learning phenomenon in which a new neuron activates at each saddle. This analysis recovers the known result of Dana et al. (2025, arXiv:2502.16977) that the network interpolates the training data with high probability as soon as $m \gtrsim \log(n)$, where $m$ is the network width and $n$ is the number of training samples. This incremental process characterization also allows us to derive a novel implicit bias result: the learned interpolator has a squared $\ell_2$-norm scaling as $\sqrt{n}$, which is within a constant factor of the minimal $\ell_2$-norm interpolator. More broadly, our work provides the first rigorous proof of an incremental learning process for ReLU networks, whilst suggesting mildly overparameterized networks can converge to interpolating solutions whose complexity is of the same order as that of the optimal interpolator.

Problem

• overparameterization
• ReLU networks
• incremental learning
• implicit bias
• gradient flow

Innovation

• incremental learning
• implicit bias
• ReLU networks
• mild overparameterization
• gradient flow