Are Graph Transformers Necessary? Efficient Long-Range Message Passing with Fractal Nodes in MPNNs

2025-11-17

AI Summary

Graph neural networks (GNNs) struggle to simultaneously achieve efficient local message passing and effective modeling of global long-range dependencies. To address this, we propose the “fractal node” mechanism: leveraging graph partitioning to induce a fractal hierarchical structure, it enables adaptive feature aggregation at subgraph levels while introducing direct long-range connections—thereby mitigating over-compression and facilitating synergistic propagation of local and global information. Crucially, this approach preserves the original message-passing neural network (MPNN) architecture and computational efficiency, without resorting to Transformer-style attention mechanisms. Empirically, our method matches or surpasses graph Transformers across multiple benchmark graph learning tasks, while retaining MPNNs’ linear time complexity and memory efficiency. This establishes a new paradigm for scalable, efficient graph representation learning.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Graph Neural Networks (GNNs) have emerged as powerful tools for learning on graph-structured data, but often struggle to balance local and global information. While graph Transformers aim to address this by enabling long-range interactions, they often overlook the inherent locality and efficiency of Message Passing Neural Networks (MPNNs). We propose a new concept called fractal nodes, inspired by the fractal structure observed in real-world networks. Our approach is based on the intuition that graph partitioning naturally induces fractal structure, where subgraphs often reflect the connectivity patterns of the full graph. Fractal nodes are designed to coexist with the original nodes and adaptively aggregate subgraph-level feature representations, thereby enforcing feature similarity within each subgraph. We show that fractal nodes alleviate the over-squashing problem by providing direct shortcut connections that enable long-range propagation of subgraph-level representations. Experiment results show that our method improves the expressive power of MPNNs and achieves comparable or better performance to graph Transformers while maintaining the computational efficiency of MPNN by improving the long-range dependencies of MPNN.

Problem

• Balancing local and global information in graph neural networks
• Addressing over-squashing problem in message passing neural networks
• Improving long-range dependencies while maintaining computational efficiency

Innovation

• Fractal nodes enable long-range message passing
• Subgraph-level feature aggregation enhances similarity
• Shortcut connections alleviate over-squashing in MPNNs