Hydra: Dual Exponentiated Memory for Multivariate Time Series Analysis

2025-11-02

AI Summary

Existing multivariate time series models suffer from three key limitations: (1) lack of temporal inductive bias, (2) neglect of cross-dimensional dependencies among variables, and (3) low efficiency in modeling long sequences. To address these, this paper proposes the Hydra dual-head memory module—the first architecture enabling joint recursive modeling across both temporal and variable dimensions. We further design a test-time pattern priority learning mechanism to enhance adaptive pattern selection. Integrating linear RNNs with a novel bi-exponential memory structure and a 2D block-wise training algorithm, our approach achieves efficient approximate training and fully parallelizable inference. Evaluated on forecasting, classification, and anomaly detection tasks, the method consistently outperforms state-of-the-art baselines, delivering significant improvements in both modeling accuracy and computational efficiency.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

In recent years, effectively modeling multivariate time series has gained significant popularity, mainly due to its wide range of applications, ranging from healthcare to financial markets and energy management. Transformers, MLPs, and linear models as the de facto backbones of modern time series models have shown promising results in single-variant and/or short-term forecasting. These models, however: (1) are permutation equivariant and so lack temporal inductive bias, being less expressive to capture the temporal dynamics; (2) are naturally designed for univariate setup, missing the inter-dependencies of temporal and variate dimensions; and/or (3) are inefficient for Long-term time series modeling. To overcome training and inference efficiency as well as the lack of temporal inductive bias, recently, linear Recurrent Neural Networks (RNNs) have gained attention as an alternative to Transformer-based models. These models, however, are inherently limited to a single sequence, missing inter-variate dependencies, and can propagate errors due to their additive nature. In this paper, we present Hydra, a by-design two-headed meta in-context memory module that learns how to memorize patterns at test time by prioritizing time series patterns that are more informative about the data. Hydra uses a 2-dimensional recurrence across both time and variate at each step, which is more powerful than mixing methods. Although the 2-dimensional nature of the model makes its training recurrent and non-parallelizable, we present a new 2D-chunk-wise training algorithm that approximates the actual recurrence with $ imes 10$ efficiency improvement, while maintaining the effectiveness. Our experimental results on a diverse set of tasks and datasets, including time series forecasting, classification, and anomaly detection show the superior performance of Hydra compared to state-of-the-art baselines.

Problem

• Addresses lack of temporal inductive bias in time series models
• Captures inter-dependencies across temporal and variate dimensions
• Improves efficiency for long-term multivariate time series modeling

Innovation

• Dual-headed in-context memory module for pattern prioritization
• 2D recurrence across time and variate dimensions
• Chunk-wise training algorithm for efficiency improvement