STD-PLM: Understanding Both Spatial and Temporal Properties of Spatial-Temporal Data with PLM

2024-07-12

AAAI Conference on Artificial Intelligence

AI Summary

Existing spatiotemporal forecasting and imputation methods are predominantly single-task, lack zero-/few-shot generalization capability, and pretrained language models (PLMs) struggle to capture high-order spatiotemporal dependencies—particularly non-pairwise, topology-aware ones. To address these limitations, we propose the first PLM framework for unified spatiotemporal data modeling. Our approach introduces a joint spatiotemporal tokenizer to unify input representations for both tasks; incorporates topology-aware node embeddings to explicitly encode spatial structure; proposes a sandglass attention mechanism (SGA) for efficient multi-scale spatiotemporal dependency learning; and designs a constrained loss function to jointly optimize forecasting and imputation. Extensive experiments on multiple benchmark datasets demonstrate state-of-the-art performance on both tasks. Notably, our method significantly outperforms existing approaches under zero- and few-shot settings, validating its strong generalization capability and robustness to data scarcity.

Technology Category

• Planning, Routing, and Scheduling: Temporal Planning
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Spatial-temporal forecasting and imputation are important for real-world intelligent systems. Most existing methods are tailored for individual forecasting or imputation tasks but are not designed for both. Additionally, they are less effective for zero-shot and few-shot learning. While pre-trained language model (PLM) have exhibited strong pattern recognition and reasoning abilities across various tasks, including few-shot and zero-shot learning, their applications in spatial-temporal data understanding has been constrained by insufficient modeling of complex correlations such as the temporal correlations, spatial connectivity, non-pairwise and high-order spatial-temporal correlations within data. In this paper, we propose STD-PLM for understanding both spatial and temporal properties of Spatial-Temporal Data with PLM, which is capable of implementing both spatial-temporal forecasting and imputation tasks. STD-PLM understands spatial-temporal correlations via explicitly designed spatial and temporal tokenizers. Topology-aware node embeddings are designed for PLM to comprehend and exploit the topology structure of data in inductive manner. Furthermore, to mitigate the efficiency issues introduced by the PLM, we design a sandglass attention module(SGA) combined with a specific constrained loss function, which significantly improves the model's efficiency while ensuring performance. Extensive experiments demonstrate that STD-PLM exhibits competitive performance and generalization capabilities across the forecasting and imputation tasks on various datasets. Moreover, STD-PLM achieves promising results on both few-shot and zero-shot tasks.

Problem

• Addresses spatial-temporal forecasting and imputation simultaneously
• Enhances zero-shot and few-shot learning for spatial-temporal data
• Improves efficiency and performance with sandglass attention module

Innovation

• Uses spatial and temporal tokenizers for correlations
• Employs topology-aware node embeddings for structure
• Introduces sandglass attention for efficiency