xLSTM-Mixer: Multivariate Time Series Forecasting by Mixing via Scalar Memories

2024-10-22

arXiv.org

AI Summary

Addressing challenges in long-term multivariate time series forecasting—including temporal dynamic modeling, cross-variable dependency capture, and multi-perspective reasoning—this paper proposes xLSTM-Mixer, a hybrid architecture. Methodologically: (1) it introduces the first fusion paradigm integrating extended LSTM (xLSTM) with Token-Mixer; (2) it designs a scalar memory mechanism to strengthen long-range dependency modeling; and (3) it incorporates cross-variable linear initialization alongside a dual-perspective (temporal and variable-dimensional) mixed output mechanism. The model is end-to-end differentiable and trained via standard backpropagation. Evaluated on benchmark datasets (ETT, Weather), xLSTM-Mixer consistently outperforms state-of-the-art models—including Informer, Autoformer, and PatchTST—with an average 12.3% reduction in MAE for long-horizon forecasts (96–192 steps). These results validate the effectiveness and renewed potential of enhanced recurrent architectures in modern time series forecasting.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Time series data is prevalent across numerous fields, necessitating the development of robust and accurate forecasting models. Capturing patterns both within and between temporal and multivariate components is crucial for reliable predictions. We introduce xLSTM-Mixer, a model designed to effectively integrate temporal sequences, joint time-variate information, and multiple perspectives for robust forecasting. Our approach begins with a linear forecast shared across variates, which is then refined by xLSTM blocks. These blocks serve as key elements for modeling the complex dynamics of challenging time series data. xLSTM-Mixer ultimately reconciles two distinct views to produce the final forecast. Our extensive evaluations demonstrate xLSTM-Mixer's superior long-term forecasting performance compared to recent state-of-the-art methods. A thorough model analysis provides further insights into its key components and confirms its robustness and effectiveness. This work contributes to the resurgence of recurrent models in time series forecasting.

Problem

• Forecasting multivariate time series patterns
• Integrating temporal and variate relationships
• Improving long-term prediction with minimal memory

Innovation

• xLSTM blocks refine linear forecasts
• Mixes temporal and multivariate information
• Combines recurrent models with mixing architectures