Graph Transformers: A Survey

2024-07-13

arXiv.org

AI Summary

Graph Transformers face fundamental challenges in modeling graph-structured data, including insufficient inductive bias, low computational efficiency, and poor generalization. To address these, this paper presents a systematic survey of recent advances and proposes the first three-dimensional taxonomy—based on depth, scalability, and pretraining strategies—for classifying Graph Transformer architectures. We distill key design principles, including graph-aware attention fusion, variants of positional encoding, subgraph sampling, and hierarchical aggregation. Furthermore, we formally identify and comprehensively define five open challenges: scalability, robustness, interpretability, dynamic graph modeling, and data diversity. The synthesized knowledge framework provides a theoretical foundation for principled model design and significantly advances practical deployment across domains such as biological network analysis, recommender systems, and molecular modeling.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Graph Mining, Social Network Analysis & Community
• Reasoning under Uncertainty: Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Graph transformers are a recent advancement in machine learning, offering a new class of neural network models for graph-structured data. The synergy between transformers and graph learning demonstrates strong performance and versatility across various graph-related tasks. This survey provides an in-depth review of recent progress and challenges in graph transformer research. We begin with foundational concepts of graphs and transformers. We then explore design perspectives of graph transformers, focusing on how they integrate graph inductive biases and graph attention mechanisms into the transformer architecture. Furthermore, we propose a taxonomy classifying graph transformers based on depth, scalability, and pre-training strategies, summarizing key principles for effective development of graph transformer models. Beyond technical analysis, we discuss the applications of graph transformer models for node-level, edge-level, and graph-level tasks, exploring their potential in other application scenarios as well. Finally, we identify remaining challenges in the field, such as scalability and efficiency, generalization and robustness, interpretability and explainability, dynamic and complex graphs, as well as data quality and diversity, charting future directions for graph transformer research.

Problem

• Surveying recent progress and challenges in graph transformer research
• Exploring integration of graph inductive biases into transformer architecture
• Addressing scalability, generalization, and interpretability challenges in graph transformers

Innovation

• Integrate graph inductive biases into transformers
• Classify models by depth scalability pretraining strategies
• Apply transformers to node edge graph level tasks