X-LogSMask: Expand Transformer for Graph-Structured Data

2026-07-01

AI Summary

This work addresses the challenge that conventional Transformer architectures, with their fully connected self-attention mechanisms, struggle to effectively capture the sparse, structured, and multi-scale interactions inherent in graph data. To overcome this limitation, the authors propose X-LogSMask, an interpretable multi-head logarithmic structural masking approach. By introducing a logarithmic transformation into structural masking for the first time, the method distributes powers of the symmetrically normalized adjacency matrix across different attention heads, enabling multi-hop information propagation within a single layer. Without modifying the underlying Transformer architecture, X-LogSMask endows each attention head with a well-defined structural receptive radius, yielding efficient and interpretable topology-aware sparse attention. The approach achieves state-of-the-art performance on 13 out of 20 node-, edge-, and graph-level benchmarks, demonstrating competitive results even with a lightweight single-layer configuration.

Abstract

Transformers have become general-purpose architectures, but their all-to-all self-attention is poorly matched to graph data, whose interactions are sparse, structured and multi-scale. Existing Graph Transformers address this mismatch through structural encodings, hybrid message-passing modules or learned attention constraints, often introducing additional complexity and limited interpretability. Here we introduce X-LogSMask, an explainable multi-head logarithmic structural mask that injects symmetrically normalized graph topology directly into attention logits. The logarithmic transform converts structural connectivity into a topology-aware gating signal, suppressing unsupported node interactions while preserving feature-dependent attention. By assigning different powers of the normalized adjacency matrix to different attention heads, X-LogSMask gives each head a defined structural radius and supports multi-hop information propagation within a single layer. We further show that a standard Transformer encoder can be interpreted as one-step message passing on a complete graph, motivating X-LogSMask as a topology-constrained alternative to unrestricted self-attention. Across 20 node-, edge- and graph-level benchmarks, Transformers equipped with X-LogSMask achieve state-of-the-art performance on 13 datasets and remain competitive in a lightweight one-layer configuration. These results show that simple, interpretable structural masks can make self-attention an effective graph-learning operator without changing the Transformer architecture. The code is available at https://github.com/LiLeyan-0120/X-LogSMask.

Problem

• Graph Transformers
• self-attention
• graph-structured data
• structural sparsity
• multi-scale interactions

Innovation

• Graph Transformer
• Structural Mask
• Logarithmic Attention
• Multi-hop Propagation
• Interpretable Self-Attention