On Explicit Super-Expressive Approximation for Neural Networks

2026-07-07

AI Summary

This work addresses the lack of explicit, non-asymptotic quantitative characterizations of the trade-off between parameter complexity and approximation error in fixed-architecture neural networks. By innovatively leveraging the Chinese Remainder Theorem as an explicit encoding mechanism, the paper achieves super-expressive approximation of Lipschitz continuous and Hölder smooth functions under fixed width and depth constraints. Specifically, for Lipschitz functions on $[0,1]^D$, a network of width $\max\{D,4\}$ and depth 5 is constructed; for functions in the class $C^{r,\gamma}_A$, a network of width $\max\{2D, D+5N+1\}$ and depth $r+9$ is designed. The study establishes the first explicit non-asymptotic bound linking model size $\mathcal{P}$ and approximation error $\varepsilon$: $\log_2 \mathcal{P} = \mathcal{O}(\varepsilon^{-2D/(r+\gamma)} \log(1/\varepsilon))$, thereby filling a critical gap in the theoretical understanding of super-expressiveness.

Abstract

In this work, we investigate the fixed-architecture neural network approximation with explicit parameter bounds and elementary activations. While prior work demonstrated super-expressive approximation using fixed-size networks, they lack quantitative and non-asymptotic characterizations of parameter magnitude with respect to the approximation error. We resolve this issue by introducing the Chinese Remainder Theorem as a constructive encoding mechanism. For Lipschitz continuous functions on $[0,1]^D$, we construct a width-$\max\{D,4\}$, depth-$5$ network with explicit parameter-error trade-offs. For Hölder-smooth functions in $C^{r,γ}_A\left([0,1]^D\right)$, our fixed network of width $\max\{2D,\ D+5N+1\}$ and depth $r + 9$ achieves the parameter magnitude $\mathcal{P}$ bounded by $\log_2 \mathcal{P}=\mathcal{O}\bigl(\varepsilon^{-2D/(r+γ)}\log(1/\varepsilon)\bigr)$. This is the dual result compared to those in the parameter-bounded and architecture-unbounded paradigm.

Problem

• super-expressive approximation
• fixed-architecture neural networks
• parameter bounds
• approximation error
• explicit construction

Innovation

• super-expressive approximation
• fixed-architecture neural networks
• Chinese Remainder Theorem
• parameter-error trade-off
• Hölder-smooth functions