Theory-to-Practice Gap for Neural Networks and Neural Operators

2025-03-23

AI Summary

This work investigates the sampling complexity of ReLU neural networks and neural operators when learning mappings belonging to specific approximation spaces, focusing on the “theory–practice gap” between theoretically optimal convergence rates and empirically attainable ones. Within a unified $L^p$ framework, we systematically extend this gap to infinite-dimensional operator learning—its first such treatment—using Bochner integration and approximation-theoretic tools to rigorously establish that the optimal sampling convergence rate is fundamentally limited by the $1/p$-order Monte Carlo rate. Our analysis encompasses mainstream architectures including DeepONet and the Fourier Neural Operator (FNO), and improves upon existing convergence upper bounds. Key contributions are: (1) a fine-grained separation of parameter complexity from sampling complexity; (2) identification of an intrinsic Monte Carlo–type convergence bottleneck in infinite-dimensional operator learning; and (3) provision of tight theoretical limits to guide principled neural operator design.

Technology Category

• Machine Learning: Learning with Manifolds
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This work studies the sampling complexity of learning with ReLU neural networks and neural operators. For mappings belonging to relevant approximation spaces, we derive upper bounds on the best-possible convergence rate of any learning algorithm, with respect to the number of samples. In the finite-dimensional case, these bounds imply a gap between the parametric and sampling complexities of learning, known as the emph{theory-to-practice gap}. In this work, a unified treatment of the theory-to-practice gap is achieved in a general $L^p$-setting, while at the same time improving available bounds in the literature. Furthermore, based on these results the theory-to-practice gap is extended to the infinite-dimensional setting of operator learning. Our results apply to Deep Operator Networks and integral kernel-based neural operators, including the Fourier neural operator. We show that the best-possible convergence rate in a Bochner $L^p$-norm is bounded by Monte-Carlo rates of order $1/p$.

Problem

• Studies sampling complexity of ReLU networks and neural operators
• Derives upper bounds on learning convergence rates
• Extends theory-to-practice gap to infinite-dimensional operator learning

Innovation

• Derives upper bounds on learning convergence rates
• Unifies theory-to-practice gap in L^p-setting
• Extends gap analysis to infinite-dimensional operator learning