Minimum Block Width for Universal Approximation by Residual Neural Networks with Inner Width One

2026-07-05

AI Summary

This study investigates the universal approximation capability of residual neural networks with inner-layer width one under LeakyReLU/ReLU activations. Employing tools from functional analysis and approximation theory, the work establishes the precise minimal block width required for universal approximation in both $L^p$ and uniform norms. It proves that a block width of at least $\max\{d_x, d_y\}$ is both necessary and sufficient. Furthermore, it provides a constructive result showing that uniform approximation can be achieved with block width $\min\{d_x + d_y, \max\{2d_x + 1, d_y\}\}$. These findings yield tight upper and lower bounds, demonstrating that networks with block widths below this threshold cannot attain universal approximation.

Abstract

In this paper, we study the universal approximation property of residual neural networks, and obtain some new results. For input and output dimensions $d_x$ and $d_y$, and LeakyReLU, ReLU, ReLU-like activation functions, the upper and lower bounds of the block width are established. To achieve $L^p$ approximation $(1\leq p <+\infty)$ on any compact domain, we show that the exact minimum block width is $\max\{d_x,d_y\}$ when the inner width is 1. Furthermore, we show that residual neural networks with block width $\min\{d_x+d_y, \max\{2d_x+1,d_y\}\}$ can achieve uniform approximation on any compact domain under the constraint that each residual branch has inner width 1. Besides, for any activation function family, we prove that residual neural networks with block width less than $\max\{d_x, d_y\}$ cannot approximate all target functions, both in the $L^p$ sense and the uniform sense, regardless of inner width.

Problem

• universal approximation
• residual neural networks
• block width
• inner width
• minimum width

Innovation

• residual neural networks
• universal approximation
• block width
• inner width
• ReLU activation