Approximation of Analytic Functions by ReLU Neural Networks with Adjustable Depth and Width

2026-07-12

AI Summary

This work investigates the approximation capability of ReLU neural networks with jointly tunable width \(N\) and depth \(L\) for infinitely smooth analytic functions. By carefully constructing networks to approximate power functions, multivariate multiplication, and polynomials, the study establishes, for the first time within a joint \((N, L)\) parameterization framework, an approximation error bound of \(O(N^{-C L^\tau})\) with constants \(C > 0\) and \(\tau > 0\). Notably, when \(N \asymp L^d\), the exponent satisfies \(\tau = 1\), substantially improving upon classical results for finitely smooth functions. This finding underscores the dominant role of depth in approximating analytic functions and reveals a novel scaling relationship between width \(N\) and \(L^d\).

Abstract

In contrast to most studies on neural network approximation theory that characterize results through a single parameter, such as the total number of network parameters, \cite{shen2020deep} pioneered the characterization of approximation rates as a joint function of the width parameter $N$ and the depth parameter $L$, thereby granting greater architectural flexibility. Existing works using the $(N,L)$-characterization focus on function classes with finite smoothness $s$, establishing a typical approximation rate of $\mathcal{O}\left(N^{-2s/d}L^{-2s/d}\right)$ with $d$ denoting the input dimension, which indicates that network depth and width play symmetric roles for these classes. In contrast, this paper establishes upper bounds for the approximation of analytic functions, which possess infinite smoothness, via ReLU networks under the $(N,L)$-characterization. Specifically, we derive approximation rates of $\mathcal{O}\left(N^{-C L^τ}\right)$, where $C>0$ is some constant and $τ>0$ is a parameter influenced by the relation between $L$ and $N$. In particular, $τ=1$ if $N$ scales roughly as $L^d$. Our findings reveal that depth plays a more critical role than width in the context of analytic function approximation. The main technical difficulty of obtaining such upper bounds lies in the trade-off between the smoothness parameters and the approximation accuracy. To overcome this difficulty, we employ refined constructions of several ReLU networks to approximate power functions, multivariate multiplication, and polynomials, which may be of independent interest.

Problem

• analytic functions
• ReLU neural networks
• approximation theory
• network depth and width
• infinite smoothness

Innovation

• ReLU neural networks
• analytic functions
• approximation theory
• network depth and width
• exponential convergence