Geometric Layer-wise Approximation Rates for Deep Networks

2026-04-22

AI Summary

This work addresses a key limitation in existing approximation theories for deep neural networks, which typically focus solely on the final output while neglecting the quantitative role of intermediate layers. The authors propose a nested network architecture with fixed width and shared mixed activations, enabling every intermediate layer to directly approximate the target function. The approximation error at each layer is explicitly controlled by the $L^p$ modulus of continuity at a corresponding geometric scale. By treating depth as a scale-dependent progressive refinement mechanism, the framework yields explicit geometric decay of layer-wise errors: for any $L^p$-integrable function, the error at layer $\ell$ is bounded by $(2d+1)$ times the modulus of continuity at scale $N^{-\ell}$; for 1-Lipschitz functions, the error decays geometrically at rate $(2d+1)N^{-\ell}$. This architecture supports adaptive refinement without reconstruction and, for the first time, provides an explicit characterization and control of intermediate-layer approximation performance.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Learning & Optimization for CV

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Depth is widely viewed as a central contributor to the success of deep neural networks, whereas standard neural network approximation theory typically provides guarantees only for the final output and leaves the role of intermediate layers largely unclear. We address this gap by developing a quantitative framework in which depth admits a precise scale-dependent interpretation. Specifically, we design a single shared mixed-activation architecture of fixed width $2dN+d+2$ and any prescribed finite depth such that each intermediate readout $Φ_\ell$ is itself an approximant to the target function $f$. For $f\in L^p([0,1]^d)$ with $p\in [1,\infty)$, the approximation error of $Φ_\ell$ is controlled by $(2d+1)$ times the $L^p$ modulus of continuity at the geometric scale $N^{-\ell}$ for all $\ell$. The estimate reduces to the geometric rate $(2d+1)N^{-\ell}$ if $f$ is $1$-Lipschitz. Our network design is inspired by multigrade deep learning, where depth serves as a progressive refinement mechanism: each new correction targets residual information at a finer scale while the earlier correction terms remain part of the later readouts, yielding a nested architecture that supports adaptive refinement without redesigning the preceding network.

Problem

• depth
• intermediate layers
• approximation theory
• scale-dependent
• geometric rate

Innovation

• geometric approximation rate
• layer-wise refinement
• mixed-activation architecture
• modulus of continuity
• nested deep networks