The Boolean Power of ReLU

2026-08-12

AI Summary

This work investigates whether the ReLU activation function strictly outperforms eventually constant activations—such as truncated ReLU—in expressing Boolean queries within graph neural networks (GNNs). By introducing two formal languages, ReLU-MPLang and Σ-MPLang, the study characterizes the classes of queries expressible over finite undirected graphs with Boolean node features. Leveraging tools from logical expressivity, GNN theory, and formal language analysis, and combining real-coefficient linear combinations with ReLU nonlinearity, the authors rigorously establish—for the first time—that ReLU-MPLang is strictly more expressive than any Σ-MPLang built from eventually constant activation functions. This result demonstrates that ReLU-GNNs are provably superior to {TrReLU, id}-GNNs for Boolean query tasks, revealing ReLU’s unique expressive power in discrete reasoning and resolving a longstanding open problem in the field.

Abstract

We prove that, on finite simple undirected graphs equipped with a single Boolean node feature, the Boolean queries expressible in $Σ$-MPLang, for any collection $Σ$ of eventually constant activation functions and with arbitrary real coefficients, form a strict subclass of the Boolean queries expressible in ReLU-MPLang. We thereby settle a recently posed open problem: whether ReLU-MPLang is more powerful than trReLU-MPLang when it comes to Boolean queries. In particular, this implies that ReLU-GNNs are strictly more expressive than {TrReLU,id}-GNNs with respect to Boolean queries on Boolean-featured graphs.

Problem

• Boolean queries
• ReLU
• expressiveness
• graph neural networks
• MPLang

Innovation

• ReLU
• Boolean queries
• expressive power
• GNNs
• MPLang