Algorithmic Foundations of Deep Learning: Complexity-Theoretic Rates and a Characterization of Universal Approximation

2026-06-25

AI Summary

This work addresses the limitations of traditional analyses of neural network expressivity, which rely on function regularity and fail to distinguish between functions of vastly different computational complexity yet similar smoothness—such as square roots and Brownian paths. Viewing neural networks as models of real-number computation, the study introduces an algorithmic complexity perspective and establishes a novel connection to real-valued circuits, characterizing approximation power through depth, width, and the number of non-zero parameters. Leveraging computability theory, it proves that any network incorporating a non-affine, non-linear activation is universally approximating, and provides necessary and sufficient conditions for this property. By integrating techniques from real-valued circuit compilation, Besov space approximation, and numerical algorithm simulation, the approach achieves breakthroughs in universal approximation of continuous functions, minimax-optimal approximation over Besov classes, and logarithmic-error approximation of holomorphic functions, while demonstrating exponential parameter efficiency over classical Lipschitz-based methods on graph shortest-path problems.

Abstract

Feedforward neural network (NN) expressivity is typically studied by emulating optimal basis-expansion schemes. While powerful, this perspective is incomplete: it primarily captures complexity through regularity, and therefore does not distinguish intuitively simple and complicated objects with comparable regularity, such as the square-root function and a typical Brownian path. The guiding message is that neural networks should be viewed not only as flexible basis functions, but also as models of computation. If a function is computable by a real-valued circuit over a prescribed elementary gate language, then it can be computed to comparable accuracy by an NN with explicit depth, width, and non-zero-parameter bounds controlled by the depth, width, gate count, and gate structure. Thus, neural-network complexity is not governed by regularity alone, but also by algorithmic complexity. We then show that any definable NN model satisfying a natural parallelization condition, allowing possibly multivariate non-linearities such as attention or layer normalization, is a universal approximator if and only if it contains a non-affine nonlinearity. The scope of our theory is illustrated by deducing universal approximation guarantees for continuous functions, minimax-optimal approximation guarantees for Besov classes, logarithmic-error complexity for holomorphic functions, and by showing that NNs can emulate numerical algorithms such as Newton-Raphson root finding and power iteration without architecture-specific arguments. Its precision is illustrated by shortest-path computation on $k$-vertex graphs: compiling the tropical dynamic-programming circuit yields NNs with O(log(1/ε)) non-zero parameters, exponentially improving in 1/ε over the generic $O(ε^{-c k^2})$ Lipschitz-approximation scale, for a constant c>0.

Problem

• neural network expressivity
• algorithmic complexity
• universal approximation
• regularity
• computational models

Innovation

• algorithmic complexity
• universal approximation
• neural network expressivity
• real-valued circuits
• non-affine nonlinearity