Generative Models for Long Time Series: Approximately Equivariant Recurrent Network Structures for an Adjusted Training Scheme

2025-05-08

AI Summary

To address the challenge that recurrent layers struggle to simultaneously capture long-range dependencies and time-shift equivariance in modeling long sequential time-series data, this paper proposes RVAE-ST—a generative model integrating a variational autoencoder with a novel equivariant recurrent architecture. Methodologically, it introduces a lightweight recurrent unit designed to approximate time-translation equivariance and employs a progressive sequence-length training strategy, substantially enhancing long-horizon temporal modeling capacity without increasing parameter count. The model is trained by maximizing the evidence lower bound (ELBO) and evaluated using the Fréchet time-series distance. Results demonstrate that RVAE-ST achieves state-of-the-art (SOTA) or competitive generative performance across multiple benchmark datasets, particularly exhibiting robustness on quasi-periodic, irregularly sampled, and partially non-stationary time series.

Technology Category

• Machine Learning: Deep Generative Models & Autoencoders
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

We present a simple yet effective generative model for time series data based on a Variational Autoencoder (VAE) with recurrent layers, referred to as the Recurrent Variational Autoencoder with Subsequent Training (RVAE-ST). Our method introduces an adapted training scheme that progressively increases the sequence length, addressing the challenge recurrent layers typically face when modeling long sequences. By leveraging the recurrent architecture, the model maintains a constant number of parameters regardless of sequence length. This design encourages approximate time-shift equivariance and enables efficient modeling of long-range temporal dependencies. Rather than introducing a fundamentally new architecture, we show that a carefully composed combination of known components can match or outperform state-of-the-art generative models on several benchmark datasets. Our model performs particularly well on time series that exhibit quasi-periodic structure,while remaining competitive on datasets with more irregular or partially non-stationary behavior. We evaluate its performance using ELBO, Fr'echet Distance, discriminative scores, and visualizations of the learned embeddings.

Problem

• Modeling long time series with recurrent layers
• Achieving approximate time-shift equivariance in sequences
• Handling quasi-periodic and irregular time series data

Innovation

• VAE with recurrent layers for time series
• Progressive training for long sequences
• Approximate time-shift equivariance design