On the Depth of Monotone ReLU Neural Networks and ICNNs

2025-05-09

AI Summary

This work investigates the expressive power of monotone ReLU networks (ReLU⁺) and input-convex neural networks (ICNNs), focusing on their ability to compute the maximum function MAXₙ and associated depth complexity separations. Methodologically, it bridges polyhedral geometry, isoperimetric properties of simplicial subdivisions, and convex analysis to recast representability as a geometric constructibility problem. Key contributions: (i) the first proof that ReLU⁺ networks neither exactly compute nor uniformly approximate MAXₙ; (ii) a tight depth lower bound of *n* for ICNNs computing MAXₙ; and (iii) a strict depth separation: any ICNN of depth *k* cannot simulate a depth-2 ReLU network of size Ω(*k*²). Collectively, these results precisely characterize the fundamental expressive limitations of ReLU⁺ and ICNN architectures under convexity constraints.

Technology Category

• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Search and Optimization: Non-convex Optimization
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Computational social science
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We study two models of ReLU neural networks: monotone networks (ReLU$^+$) and input convex neural networks (ICNN). Our focus is on expressivity, mostly in terms of depth, and we prove the following lower bounds. For the maximum function MAX$_n$ computing the maximum of $n$ real numbers, we show that ReLU$^+$ networks cannot compute MAX$_n$, or even approximate it. We prove a sharp $n$ lower bound on the ICNN depth complexity of MAX$_n$. We also prove depth separations between ReLU networks and ICNNs; for every $k$, there is a depth-2 ReLU network of size $O(k^2)$ that cannot be simulated by a depth-$k$ ICNN. The proofs are based on deep connections between neural networks and polyhedral geometry, and also use isoperimetric properties of triangulations.

Problem

• Study depth expressivity of ReLU+ and ICNN models
• Prove lower bounds for MAXn computation in ReLU+ networks
• Establish depth separations between ReLU networks and ICNNs

Innovation

• ReLU$^+$ networks cannot compute MAX$_n$
• ICNN depth complexity bound for MAX$_n$
• Depth separations between ReLU and ICNN