Convergence Guarantees of Gradient Descent for Neural Networks via Generalized Lipschitz Smoothness

2026-08-11

AI Summary

This work establishes convergence guarantees for gradient descent in feedforward neural networks of arbitrary width and depth, without requiring special initialization schemes or data assumptions. By introducing a generalized Lipschitz smoothness condition, the authors characterize the structural properties of the composition between common activation functions—such as tanh and sigmoid—and the mean squared error loss, showing that multi-layer compositions partially retain Lipschitz continuity. Leveraging parameter norm control and a descent lemma, the theoretical analysis demonstrates that for an L-layer network, the minimum gradient norm over T iterations converges to zero at a rate of O(1/T^{1/L}). This result provides the first characterization of the global convergence rate of gradient descent for deep networks under general conditions.

Abstract

We establish convergence guarantees of gradient descent for general feedforward neural networks of arbitrary width or depth, with no special requirements on the initialization or dataset. We only assume that the activation functions are Lipschitz smooth, Lipschitz continuous, and linearly bounded--- properties that hold for linear, tanh, softplus, and sigmoid activation functions. For the loss function, we require that it is Lipschitz smooth in the model outputs, which is true for mean-squared error. The key theoretical insight is that the Lipschitz properties of the activation functions are partially preserved even through repeated compositions, leading to a novel generalized Lipschitz smoothness condition where the change in gradient is upper bounded by the change in the parameter space, multiplied by polynomial terms of the parameter norms at both endpoints. This type of condition holds for both the model function and the loss function, enabling a descent lemma where the loss decreases as long as the learning rate is small enough with respect to the parameter norms. By ensuring that the parameter norms do not grow too quickly to infinity, we prove that the minimum squared gradient norm converges to zero in $T$ iterations at rate $O(1/T^{1/L})$ for an $L$-layer neural network.

Problem

• convergence guarantees
• gradient descent
• neural networks
• Lipschitz smoothness
• feedforward networks

Innovation

• generalized Lipschitz smoothness
• gradient descent convergence
• feedforward neural networks
• activation function properties
• descent lemma