Transformers are Graph Neural Networks

2025-06-27

AI Summary

This work investigates the theoretical connections between Transformers and Graph Neural Networks (GNNs), challenging the conventional view of Transformers as purely sequence-based models. Method: We formalize the Transformer as a message-passing GNN operating on a fully connected token graph, where self-attention implements dynamic, content-aware neighborhood aggregation and positional encodings implicitly encode structural priors. We reinterpret Transformer computation within a unified message-passing framework and introduce the concept of the “hardware lottery”—highlighting that its efficiency stems not only from architectural design but also from hardware-level optimizations for dense matrix operations in modern accelerators. Contribution/Results: (1) We establish the first rigorous theoretical bridge between NLP and graph representation learning; (2) we identify the dual origins of Transformer expressivity—structural modeling capacity and hardware alignment; and (3) we propose a new paradigm for interpretable modeling and hardware-aware neural architecture design.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: (Large) Language Models
• Computer Vision: Generative Adversarial Networks (GANs) for Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We establish connections between the Transformer architecture, originally introduced for natural language processing, and Graph Neural Networks (GNNs) for representation learning on graphs. We show how Transformers can be viewed as message passing GNNs operating on fully connected graphs of tokens, where the self-attention mechanism capture the relative importance of all tokens w.r.t. each-other, and positional encodings provide hints about sequential ordering or structure. Thus, Transformers are expressive set processing networks that learn relationships among input elements without being constrained by apriori graphs. Despite this mathematical connection to GNNs, Transformers are implemented via dense matrix operations that are significantly more efficient on modern hardware than sparse message passing. This leads to the perspective that Transformers are GNNs currently winning the hardware lottery.

Problem

• Connecting Transformers and Graph Neural Networks
• Viewing Transformers as message passing GNNs
• Transformers' efficiency due to dense matrix operations

Innovation

• Transformers as message passing GNNs
• Self-attention captures token relationships
• Dense matrix operations enhance efficiency