Breaking the Context Bottleneck on Long Time Series Forecasting

2024-12-21

arXiv.org

AI Summary

Long-term time series forecasting faces challenges of overfitting and non-stationarity induced by long input sequences, prompting existing methods to truncate input length to curb error accumulation. To address this, we propose the LogSparse Decomposition Multi-scale (LDM) framework—a novel architecture integrating multi-scale decomposition with LogSparse attention. Hierarchical decomposition explicitly disentangles trend, seasonal, and residual components across scales, substantially mitigating non-stationarity; meanwhile, LogSparse attention enables compact representation learning and architectural lightweighting. Evaluated on major long-horizon benchmarks—including ETT, Weather, and Electricity—LDM achieves state-of-the-art performance: average prediction error reduced by 12.6%, training memory consumption decreased by 38%, and inference speed accelerated by 2.1×. The framework thus delivers superior accuracy, computational efficiency, and generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Long-term time-series forecasting is essential for planning and decision-making in economics, energy, and transportation, where long foresight is required. To obtain such long foresight, models must be both efficient and effective in processing long sequence. Recent advancements have enhanced the efficiency of these models; however, the challenge of effectively leveraging longer sequences persists. This is primarily due to the tendency of these models to overfit when presented with extended inputs, necessitating the use of shorter input lengths to maintain tolerable error margins. In this work, we investigate the multiscale modeling method and propose the Logsparse Decomposable Multiscaling (LDM) framework for the efficient and effective processing of long sequences. We demonstrate that by decoupling patterns at different scales in time series, we can enhance predictability by reducing non-stationarity, improve efficiency through a compact long input representation, and simplify the architecture by providing clear task assignments. Experimental results demonstrate that LDM not only outperforms all baselines in long-term forecasting benchmarks, but also reducing both training time and memory costs.

Problem

• Overcoming overfitting in long sequence time series forecasting
• Enhancing efficiency and effectiveness in processing extended inputs
• Reducing non-stationarity through multiscale pattern decoupling

Innovation

• Multiscale modeling decouples patterns at different scales
• Compact long input representation improves efficiency
• Clear task assignments simplify architecture