On the algorithmic construction of deep ReLU networks

2025-06-23

AI Summary

This work investigates the **exact functional representational capacity** of deep ReLU feedforward neural networks: which classical algorithms can be explicitly constructed as piecewise-linear functions? Departing from conventional black-box training paradigms, we treat neural networks as **programmable mathematical objects**, adopting an algorithmic construction perspective. We present the first explicit neural implementation of **exact sorting of input sequences**—a canonical nonlinear, nonsmooth computational task. Our architecture integrates recursive structures with parallel computation modules, yielding deep networks with billions of parameters and near-optimal time complexity—O(n log n). We provide rigorous theoretical analysis showing that, for a fixed parameter budget, deep networks achieve strictly superior representational efficiency over shallow networks for structured algorithms like sorting. This establishes a fundamental advantage of depth in encoding complex algorithmic logic via piecewise-linear representations.

Technology Category

• Machine Learning: Deep Learning Algorithms
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

It is difficult to describe in mathematical terms what a neural network trained on data represents. On the other hand, there is a growing mathematical understanding of what neural networks are in principle capable of representing. Feedforward neural networks using the ReLU activation function represent continuous and piecewise linear functions and can approximate many others. The study of their expressivity addresses the question: which ones? Contributing to the available answers, we take the perspective of a neural network as an algorithm. In this analogy, a neural network is programmed constructively, rather than trained from data. An interesting example is a sorting algorithm: we explicitly construct a neural network that sorts its inputs exactly, not approximately, and that, in a sense, has optimal computational complexity if the input dimension is large. Such constructed networks may have several billion parameters. We construct and analyze several other examples, both existing and new. We find that, in these examples, neural networks as algorithms are typically recursive and parallel. Compared to conventional algorithms, ReLU networks are restricted by having to be continuous. Moreover, the depth of recursion is limited by the depth of the network, with deep networks having superior properties over shallow ones.

Problem

• Understanding mathematical representation of trained ReLU networks
• Constructing exact sorting algorithms via neural networks
• Exploring recursive and parallel properties in network algorithms

Innovation

• Constructs exact sorting networks using ReLU
• Employs recursive and parallel network designs
• Leverages deep networks for superior performance