Expressive Power of Graph Transformers via Logic

2025-08-01

AI Summary

This work systematically characterizes the expressive power boundaries of Graph Transformers (GTs) and GPS networks under soft and average-hard attention mechanisms. Addressing both idealized (real-number) and practical (floating-point) numerical settings, we establish the first precise logical correspondences: GPS networks are expressively equivalent to Graded Modal Logic (GML) with global modalities over reals, and to its counting extension (CGML) with counting global modalities over floats; GTs correspond instead to propositional logic extended with global modalities in both settings. Leveraging tools from first-order, propositional, and modal logic, we rigorously expose how numerical precision fundamentally constrains logical expressivity. Our framework provides the first fine-grained, formally verifiable logical characterization of graph neural networks, enabling principled analysis of their representational capabilities.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Transformers are the basis of modern large language models, but relatively little is known about their precise expressive power on graphs. We study the expressive power of graph transformers (GTs) by Dwivedi and Bresson (2020) and GPS-networks by Rampásek et al. (2022), both under soft-attention and average hard-attention. Our study covers two scenarios: the theoretical setting with real numbers and the more practical case with floats. With reals, we show that in restriction to vertex properties definable in first-order logic (FO), GPS-networks have the same expressive power as graded modal logic (GML) with the global modality. With floats, GPS-networks turn out to be equally expressive as GML with the counting global modality. The latter result is absolute, not restricting to properties definable in a background logic. We also obtain similar characterizations for GTs in terms of propositional logic with the global modality (for reals) and the counting global modality (for floats).

Problem

• Study expressive power of graph transformers and GPS-networks
• Compare theoretical real numbers vs practical float scenarios
• Characterize expressive power in terms of logic modalities

Innovation

• Study expressive power of graph transformers
• Compare GPS-networks with graded modal logic
• Characterize GTs using propositional logic variants