Towards Efficient Large Scale Spatial-Temporal Time Series Forecasting via Improved Inverted Transformers

2025-03-13

AI Summary

To address scalability bottlenecks and high computational complexity arising from dynamic entity addition/removal and explosive growth in large-scale spatiotemporal time series forecasting, this paper proposes EiFormer—a linear-complexity Transformer architecture. Methodologically, EiFormer fundamentally restructures the attention mechanism to eliminate redundant computations, introduces an enhanced inverted Transformer design, and integrates linear attention with randomized projection to strengthen feature representation and generalization capability. Extensive experiments on the LargeST benchmark and a real-world global payment network industrial dataset demonstrate that EiFormer achieves O(N) training and inference complexity—reducing runtime by multiple folds over state-of-the-art methods—while simultaneously delivering significantly improved prediction accuracy. By reconciling efficiency with precision, EiFormer establishes a scalable, practical paradigm for ultra-large-scale synchronous spatiotemporal forecasting.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Time series forecasting at scale presents significant challenges for modern prediction systems, particularly when dealing with large sets of synchronized series, such as in a global payment network. In such systems, three key challenges must be overcome for accurate and scalable predictions: 1) emergence of new entities, 2) disappearance of existing entities, and 3) the large number of entities present in the data. The recently proposed Inverted Transformer (iTransformer) architecture has shown promising results by effectively handling variable entities. However, its practical application in large-scale settings is limited by quadratic time and space complexity ($O(N^2)$) with respect to the number of entities $N$. In this paper, we introduce EiFormer, an improved inverted transformer architecture that maintains the adaptive capabilities of iTransformer while reducing computational complexity to linear scale ($O(N)$). Our key innovation lies in restructuring the attention mechanism to eliminate redundant computations without sacrificing model expressiveness. Additionally, we incorporate a random projection mechanism that not only enhances efficiency but also improves prediction accuracy through better feature representation. Extensive experiments on the public LargeST benchmark dataset and a proprietary large-scale time series dataset demonstrate that EiFormer significantly outperforms existing methods in both computational efficiency and forecasting accuracy. Our approach enables practical deployment of transformer-based forecasting in industrial applications where handling time series at scale is essential.

Problem

• Addresses challenges in large-scale time series forecasting.
• Reduces computational complexity of Inverted Transformers to linear scale.
• Improves prediction accuracy and efficiency in industrial applications.

Innovation

• Improved inverted transformer reduces complexity
• Restructured attention mechanism eliminates redundant computations
• Random projection enhances efficiency and accuracy