Low-rank Adaptation for Spatio-Temporal Forecasting

2024-04-11

arXiv.org

AI Summary

Existing spatiotemporal forecasting models often neglect node-level heterogeneity, while node-specific modeling tends to cause over-parameterization. To address this, we propose ST-LoRA—a plug-and-play low-rank adaptation framework for spatiotemporal prediction that requires no modification to backbone architectures. Our approach introduces: (1) node-adaptive low-rank layers to explicitly capture regional heterogeneity; and (2) a multi-layer residual fusion stacking module to enhance feature representation. Built upon low-rank matrix decomposition and node-adaptive parameterization, ST-LoRA achieves lightweight integration (<4% increase in parameters and training cost), broad compatibility, and cross-dataset transferability. Extensive experiments across six real-world traffic datasets and six state-of-the-art base models demonstrate consistent and significant accuracy improvements, validating its effectiveness, robustness, and generalizability.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Machine Learning: Mixture of Experts (MoE)
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Spatio-temporal forecasting is crucial in real-world dynamic systems, predicting future changes using historical data from diverse locations. Existing methods often prioritize the development of intricate neural networks to capture the complex dependencies of the data, yet their accuracy fails to show sustained improvement. Besides, these methods also overlook node heterogeneity, hindering customized prediction modules from handling diverse regional nodes effectively. In this paper, our goal is not to propose a new model but to present a novel low-rank adaptation framework as an off-the-shelf plugin for existing spatial-temporal prediction models, termed ST-LoRA, which alleviates the aforementioned problems through node-level adjustments. Specifically, we first tailor a node adaptive low-rank layer comprising multiple trainable low-rank matrices. Additionally, we devise a multi-layer residual fusion stacking module, injecting the low-rank adapters into predictor modules of various models. Across six real-world traffic datasets and six different types of spatio-temporal prediction models, our approach minimally increases the parameters and training time of the original models by less than 4%, still achieving consistent and sustained performance enhancement.

Problem

• Addresses node-level heterogeneity in spatio-temporal forecasting
• Reduces over-parameterization in modeling node-specific characteristics
• Enhances performance with minimal computational overhead

Innovation

• Node-adaptive low-rank layer for efficiency
• Node-specific predictor captures heterogeneity
• Minimal overhead with 1% extra parameters