Depth Lower Bounds for ReLU Networks with Binary Inputs

2026-06-16

AI Summary

This work investigates the role of depth in the expressive power of ReLU neural networks when processing binary inputs. By constructing a class of functions that can be computed exactly by ReLU networks of depth \(n+1\) and constant width, the authors prove that any ReLU network of depth \(d\) and width \(w\) capable of exactly representing such a function must satisfy \(w^d = \Omega(2^n)\). This result establishes, for the first time, a depth-separation lower bound over the Boolean hypercube \(\{0,1\}^n\) that holds across all depth regimes. It highlights the critical importance of depth for exact computation and demonstrates that when \(d = o(n / \log n)\), no polynomial-width network in \(n\) can achieve exact representation, thereby revealing an exponential trade-off between depth and width.

Abstract

We study the role of depth in ReLU networks with discrete (Boolean) inputs and real-valued outputs, complementing two established lines of work. For Boolean inputs, striking depth separation results were proven for $\mathsf{AC}^0$ but with threshold ($\mathsf{TC}^0$) or ReLU gates depth separation is only established for depth two vs. three. On the other hand, for {\em real-valued} functions and ReLU networks, Telgarsky's (2016) constructed a simple one variable class of functions which establishes separation at higher depths. In this paper we are interested to establish an all-depths depth separation for ReLU networks on $\{0,1\}^n$. We do so by exhibiting an explicit family of functions computable exactly by a ReLU network of depth $n+1$ and constant width, such that any ReLU network of depth $d$ and width $w$ computing the function exactly must satisfy $w^d = Ω(2^n)$; in particular, no network of depth $d = o(n/\log n)$ can compute it with width polynomial in $n$. We note that our lower bound relies on \emph{exact, infinite-accuracy} computation as an exponential precision truncation of the output is computable by a polynomial-size $\mathsf{TC}^0$ circuit.

Problem

• depth separation
• ReLU networks
• Boolean inputs
• neural network depth
• computational complexity

Innovation

• depth separation
• ReLU networks
• Boolean inputs
• circuit complexity
• exact computation