Dense Neural Networks are not Universal Approximators

2026-02-07

AI Summary

This study investigates whether densely connected neural networks retain universal approximation capability under practical constraints on weight magnitude and network width. By integrating insights from model compression, a weak regularity lemma, and the message-passing perspective of graph neural networks, we rigorously analyze the approximation limits of dense ReLU networks for Lipschitz continuous functions. We establish, for the first time under natural constraints, that such restricted dense networks cannot approximate all Lipschitz functions, thereby disproving their universal approximation property. Our findings further reveal that sparse connectivity is essential for achieving universal approximation, offering a theoretical explanation for the expressive limitations of dense architectures under realistic resource constraints.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Social networks and social learning
• Social Networks and Social Media: Social media analysis through the lenses of networks

Abstract

We investigate the approximation capabilities of dense neural networks. While universal approximation theorems establish that sufficiently large architectures can approximate arbitrary continuous functions if there are no restrictions on the weight values, we show that dense neural networks do not possess this universality. Our argument is based on a model compression approach, combining the weak regularity lemma with an interpretation of feedforward networks as message passing graph neural networks. We consider ReLU neural networks subject to natural constraints on weights and input and output dimensions, which model a notion of dense connectivity. Within this setting, we demonstrate the existence of Lipschitz continuous functions that cannot be approximated by such networks. This highlights intrinsic limitations of neural networks with dense layers and motivates the use of sparse connectivity as a necessary ingredient for achieving true universality.

Problem

• dense neural networks
• universal approximation
• Lipschitz continuous functions
• model compression
• sparse connectivity

Innovation

• dense neural networks
• universal approximation
• model compression
• weak regularity lemma
• sparse connectivity