Gradient Descent Finds Over-Parameterized Neural Networks with Sharp Generalization for Nonparametric Regression: A Distribution-Free Analysis

2024-11-05

arXiv.org

AI Summary

This paper investigates the generalization performance of overparameterized two-layer neural networks in nonparametric regression. Addressing the limitation of existing theory—which relies on strong distributional assumptions on covariates (e.g., spherical uniformity)—we propose a distribution-free analytical framework: under only boundedness (not distributional) assumptions on covariates, we rigorously characterize the required network width lower bound and optimal early-stopping time when employing constant-step-size gradient descent with early stopping. By integrating Neural Tangent Kernel (NTK) analysis, nonparametric statistics, and generalization error bounds, we establish for the first time that the trained network achieves the minimax-optimal convergence rate $O(varepsilon_n^2)$, matching that of classical kernel regression. This result bridges the theoretical gap between the infinite-width NTK limit and finite-width practical training, providing critical foundational support for the nonparametric statistical understanding of deep learning.

Technology Category

• Machine Learning: Learning Theory
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Non-convex Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We study nonparametric regression by an over-parameterized two-layer neural network trained by gradient descent (GD) in this paper. We show that, if the neural network is trained by GD with early stopping, then the trained network renders a sharp rate of the nonparametric regression risk of $cO(eps_n^2)$, which is the same rate as that for the classical kernel regression trained by GD with early stopping, where $eps_n$ is the critical population rate of the Neural Tangent Kernel (NTK) associated with the network and $n$ is the size of the training data. It is remarked that our result does not require distributional assumptions about the covariate as long as the covariate is bounded, in a strong contrast with many existing results which rely on specific distributions of the covariates such as the spherical uniform data distribution or distributions satisfying certain restrictive conditions. The rate $cO(eps_n^2)$ is known to be minimax optimal for specific cases, such as the case that the NTK has a polynomial eigenvalue decay rate which happens under certain distributional assumptions on the covariates. Our result formally fills the gap between training a classical kernel regression model and training an over-parameterized but finite-width neural network by GD for nonparametric regression without distributional assumptions on the bounded covariate. We also provide confirmative answers to certain open questions or address particular concerns in the literature of training over-parameterized neural networks by GD with early stopping for nonparametric regression, including the characterization of the stopping time, the lower bound for the network width, and the constant learning rate used in GD.

Problem

• Analyzing generalization of over-parameterized neural networks trained by gradient descent
• Comparing neural network regression risk to classical kernel regression rates
• Removing distributional assumptions for covariates in nonparametric regression

Innovation

• Over-parameterized two-layer neural network
• Gradient descent with early stopping
• No distributional assumptions on covariates