Optimal Learning Rate Scaling Depends on Data in Deep Scalar Linear Networks

2026-07-08

AI Summary

This work investigates whether learning rate scaling rules in deep scalar linear networks depend on the data distribution and examines the generalization of universal scaling strategies across network depths. Leveraging the fact that gradient descent dynamics in this model admit exact analytical solutions, the authors conduct a theoretical analysis incorporating residual connections and demonstrate that the optimal learning rate scaling is fundamentally determined by the data distribution. Under this data-dependent optimal scaling, the learning dynamics become independent of the specific data realization and achieve a constant linear convergence rate at any depth—including infinite depth—substantially outperforming data-agnostic scaling strategies.

Abstract

In this short note we consider the gradient descent dynamics of deep scalar linear networks, $f(x) = \prod_{l=1}^L w_l x$, which enjoy exact time-course solutions for any integer depth. We show that even in this minimal model, the optimal depth-wise learning rate scaling depends on data, whereas data-agnostic scaling rules fail to transfer across depths. Under the data-dependent optimal scaling, the learning dynamics is independent of data and weakly dependent on depth, resulting in a constant linear convergence rate across all depths including infinity. We further show similar data-dependent effects in deep scalar linear networks with residual connections.

Problem

• learning rate scaling
• deep scalar linear networks
• data dependence
• gradient descent dynamics
• residual connections

Innovation

• learning rate scaling
• data-dependent optimization
• deep linear networks
• gradient descent dynamics
• residual connections