ST-LINK: Spatially-Aware Large Language Models for Spatio-Temporal Forecasting

2025-09-17

AI Summary

Large language models (LLMs) face challenges in graph-structured spatiotemporal forecasting tasks—such as traffic flow prediction—due to their sequence-centric architecture, which inadequately captures topological dependencies among nodes. To address this, we propose SE-Attention: a novel mechanism that injects spatial adjacency information into rotary position embeddings and integrates memory-augmented graph attention with a dynamic historical pattern retrieval module (MRFFN). This enables end-to-end spatiotemporal modeling with explicit spatial awareness while preserving the original LLM’s sequential backbone and standard training paradigm—requiring no architectural modifications. Evaluated on multiple real-world traffic datasets, our method consistently outperforms state-of-the-art graph neural networks and LLM-based baselines, achieving superior long-horizon forecasting accuracy under both regular trends and sudden anomalies.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Traffic forecasting represents a crucial problem within intelligent transportation systems. In recent research, Large Language Models (LLMs) have emerged as a promising method, but their intrinsic design, tailored primarily for sequential token processing, introduces notable challenges in effectively capturing spatial dependencies. Specifically, the inherent limitations of LLMs in modeling spatial relationships and their architectural incompatibility with graph-structured spatial data remain largely unaddressed. To overcome these limitations, we introduce ST-LINK, a novel framework that enhances the capability of Large Language Models to capture spatio-temporal dependencies. Its key components are Spatially-Enhanced Attention (SE-Attention) and the Memory Retrieval Feed-Forward Network (MRFFN). SE-Attention extends rotary position embeddings to integrate spatial correlations as direct rotational transformations within the attention mechanism. This approach maximizes spatial learning while preserving the LLM's inherent sequential processing structure. Meanwhile, MRFFN dynamically retrieves and utilizes key historical patterns to capture complex temporal dependencies and improve the stability of long-term forecasting. Comprehensive experiments on benchmark datasets demonstrate that ST-LINK surpasses conventional deep learning and LLM approaches, and effectively captures both regular traffic patterns and abrupt changes.

Problem

• Enhancing LLMs to capture spatial dependencies in traffic forecasting
• Addressing architectural incompatibility with graph-structured spatial data
• Improving long-term forecasting stability and complex temporal pattern capture

Innovation

• Spatially-Enhanced Attention for spatial correlations
• Rotary position embeddings with rotational transformations
• Memory Retrieval Feed-Forward Network for temporal patterns