Language Model Empowered Spatio-Temporal Forecasting via Physics-Aware Reprogramming

2024-08-24

AI Summary

To address the challenge of modeling complex spatiotemporal correlations in numerical sequences under data-scarce conditions—where conventional pre-trained language models (PLMs) struggle—this paper proposes RePST, a physics-aware spatiotemporal reprogramming framework. RePST introduces two key innovations: (1) a physics-guided temporal decomposition module that enables interpretable disentanglement of spatiotemporal dynamics, and (2) a selective discrete reprogramming mechanism that preserves fidelity in discrete sequence representation. By extending the spatiotemporal vocabulary and adapting PLMs—including LLaMA and BERT—to physics-informed modeling, RePST bridges the semantic gap between general-purpose PLMs and domain-specific physical constraints. Extensive experiments on multiple real-world spatiotemporal datasets demonstrate that RePST consistently outperforms 12 state-of-the-art methods, achieving an average 18.7% reduction in prediction error under few-shot settings, while significantly improving generalization and robustness.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Planning, Routing, and Scheduling: Temporal Planning
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Psychology-informed user models and recommender systems

Abstract

Spatio-temporal forecasting is pivotal in numerous real-world applications, including transportation planning, energy management, and climate monitoring. In this work, we aim to harness the reasoning and generalization abilities of Pre-trained Language Models (PLMs) for more effective spatio-temporal forecasting, particularly in data-scarce scenarios. However, recent studies uncover that PLMs, which are primarily trained on textual data, often falter when tasked with modeling the intricate correlations in numerical time series, thereby limiting their effectiveness in comprehending spatio-temporal data. To bridge the gap, we propose RePST, a physics-aware PLM reprogramming framework tailored for spatio-temporal forecasting. Specifically, we first propose a physics-aware decomposer that adaptively disentangles spatially correlated time series into interpretable sub-components, which facilitates PLM to understand sophisticated spatio-temporal dynamics via a divide-and-conquer strategy. Moreover, we propose a selective discrete reprogramming scheme, which introduces an expanded spatio-temporal vocabulary space to project spatio-temporal series into discrete representations. This scheme minimizes the information loss during reprogramming and enriches the representations derived by PLMs. Extensive experiments on real-world datasets show that the proposed RePST outperforms twelve state-of-the-art baseline methods, particularly in data-scarce scenarios, highlighting the effectiveness and superior generalization capabilities of PLMs for spatio-temporal forecasting.

Problem

• Enhancing spatio-temporal forecasting using pre-trained language models
• Overcoming PLM limitations in modeling numerical time series correlations
• Addressing data scarcity in spatio-temporal forecasting tasks

Innovation

• Semantic-oriented PLM reprogramming framework
• Adaptive disentangling of time series
• Selective discrete reprogramming scheme