Graph Masked Language Models

2025-02-24

AI Summary

This work addresses the challenge of effectively fusing graph-structured and textual information. We propose a dual-branch collaborative learning framework: one branch employs a Graph Neural Network (GNN) to model graph topology, while the other leverages a Pre-trained Language Model (PLM) to capture textual semantics. Our key innovation is a topology-aware semantic masking strategy—dynamically selecting masking targets based on node-theoretic importance—and a learnable soft masking mechanism (feature interpolation) that enables fine-grained alignment between structural and semantic representations. Furthermore, we introduce joint contrastive learning to optimize cross-modal embeddings. Evaluated on multiple node classification and language understanding benchmarks, our method achieves state-of-the-art performance, with significant improvements in model robustness, training convergence stability, and cross-task generalization capability.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Language Models (LMs) and Graph Neural Networks (GNNs) have shown great promise in their respective areas, yet integrating structured graph data with rich textual information remains challenging. In this work, we propose emph{Graph Masked Language Models} (GMLM), a novel dual-branch architecture that combines the structural learning of GNNs with the contextual power of pretrained language models. Our approach introduces two key innovations: (i) a emph{semantic masking strategy} that leverages graph topology to selectively mask nodes based on their structural importance, and (ii) a emph{soft masking mechanism} that interpolates between original node features and a learnable mask token, ensuring smoother information flow during training. Extensive experiments on multiple node classification and language understanding benchmarks demonstrate that GMLM not only achieves state-of-the-art performance but also exhibits enhanced robustness and stability. This work underscores the benefits of integrating structured and unstructured data representations for improved graph learning.

Problem

• Integrating structured graph data with textual information effectively
• Combining GNN structural learning with pretrained language models' power
• Improving graph learning via structured and unstructured data fusion

Innovation

• Dual-branch architecture combining GNNs and LMs
• Semantic masking strategy using graph topology
• Soft masking mechanism for smoother training