Formalizing and Mitigating Structural Distortion in LLM Attention for Zero-Shot Graph Reasoning

2026-06-14

AI Summary

This work addresses the degradation of zero-shot graph reasoning performance in large language models caused by attention attenuation when processing graph-structured data through linearization, where neighboring nodes become distant in the input sequence. The study formally characterizes this attention distortion mechanism for the first time, revealing a direct relationship between graph bandwidth and attention decay under rotary positional encoding. To mitigate this issue, the authors propose GaLA, a lightweight, training-free, plug-and-play correction method that guides attention toward adjacent nodes during inference while preserving the original sequential inductive bias. Extensive experiments demonstrate that GaLA consistently yields significant improvements in zero-shot performance across multiple textual attributed graph benchmarks with negligible computational overhead, confirming structural distortion as a critical yet correctable bottleneck.

Abstract

Large Language Models (LLMs) have shown promise for reasoning over Text-Attributed Graphs (TAGs). However, applying LLMs to graphs requires linearizing their structure into sequences, introducing distortion rooted in the graph bandwidth problem. While this distortion has been shown to degrade performance, it is often attributed to prompt design or model scale, leaving the underlying mechanism unclear. In this work, we show \textit{how} rotary positional embeddings turn graph linearization into bandwidth-dependent attention decay, suppressing attention between graph-adjacent nodes that are forced far apart in the serialized sequence. This shifts the focus of LLM-based graph reasoning from prompt engineering and scaling toward correcting attention misalignment. Motivated by this analysis, we propose \textbf{G}raph-\textbf{a}ligned \textbf{L}anguage \textbf{A}ttention (\textbf{GaLA}), a lightweight, inference-time modification for LLMs. GaLA biases attention toward graph-adjacent nodes while preserving the LLM's sequential inductive biases. Across TAG benchmarks, GaLA improves performance with negligible overhead, demonstrating that distortion is a correctable bottleneck in LLM-based graph reasoning.

Problem

• structural distortion
• graph linearization
• attention decay
• zero-shot graph reasoning
• text-attributed graphs

Innovation

• structural distortion
• graph linearization
• rotary positional embeddings
• attention alignment
• zero-shot graph reasoning