Recurrent neural networks approximate continuous functions

2026-06-18

AI Summary

This work demonstrates that a ReLU recurrent neural network (RNN) with fixed weights and hidden dimension can, through extended runtime, uniformly approximate any continuous function on $[-1, 1]$, overcoming a key limitation of conventional neural networks whose expressivity is constrained by static architectures. To establish this result, the authors introduce the Turing Machine with Neural Units (TMNU) as an intermediate computational model that retains algorithmic flexibility while being explicitly simulatable by such an RNN. The proposed framework achieves uniform approximation at a rate matching that of underlying polynomial approximations, and a minimax lower bound analysis rigorously establishes runtime as an essential computational resource within this paradigm.

Abstract

Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [-1,1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.

Problem

• recurrent neural networks
• function approximation
• fixed architecture
• runtime complexity
• continuous functions

Innovation

• recurrent neural networks
• fixed architecture approximation
• Turing machine with neural units
• uniform approximation
• runtime complexity