Quantitative Gaussian-Process limits of Tensor Programs

2026-07-07

AI Summary

This work investigates the limiting behavior of randomly initialized neural networks as their width tends to infinity, wherein they converge to Gaussian processes, and establishes a quantitative error bound between the outputs of finite-width networks and their Gaussian process limits. Leveraging the tensor program framework together with Gaussian process theory and Wasserstein distance analysis, the study provides the first explicit finite-width error bounds for a broad class of architectures, including feedforward, recurrent, and Transformer-based models. The key contribution lies in developing an architecture-agnostic quantitative convergence theory, proving that the approximation error decays at a rate proportional to the inverse square root of the network width, thereby rigorously characterizing the accuracy with which finite-width networks approximate their infinite-width Gaussian process counterparts.

Abstract

We study the infinite-width Gaussian-process limit of random neural networks through the lens of tensor programs, and we provide a quantitative convergence theory in Wasserstein distance. Our main result gives explicit finite-width error bounds, of order inverse square-root of the widths between finite-network executions and their Gaussian-process limits. The framework is architecture-agnostic and covers feed-forward models together with weight-sharing schemes relevant for recurrent and transformer-type architectures.

Problem

• Gaussian-process limit
• tensor programs
• infinite-width
• random neural networks
• Wasserstein distance

Innovation

• Gaussian-process limit
• Tensor Programs
• quantitative convergence
• Wasserstein distance
• architecture-agnostic