TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation

2024-10-08

AI Summary

Existing time-series self-supervised learning methods struggle to simultaneously model long-term dynamics and capture fine-grained local patterns. To address this, we propose an autoregressive pretraining framework that integrates causal Transformers with denoising diffusion: causal Transformers enable long-range dependency modeling, while block-wise embedding and an autoregressive diffusion process—comprising forward noise injection and reverse denoising—jointly optimize global trend and local structural representations. This work is the first to unify global causal modeling and local fine-grained reconstruction within a single self-supervised objective, enhancing representation discriminability and robustness through end-to-end joint training. Extensive experiments on multiple time-series forecasting and classification benchmarks demonstrate significant improvements over state-of-the-art methods, validating the framework’s strong transferability and generalization capability.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Self-supervised learning has garnered increasing attention in time series analysis for benefiting various downstream tasks and reducing reliance on labeled data. Despite its effectiveness, existing methods often struggle to comprehensively capture both long-term dynamic evolution and subtle local patterns in a unified manner. In this work, we propose TimeDART, a novel self-supervised time series pre-training framework that unifies two powerful generative paradigms to learn more transferable representations. Specifically, we first employ a causal Transformer encoder, accompanied by a patch-based embedding strategy, to model the evolving trends from left to right. Building on this global modeling, we further introduce a denoising diffusion process to capture fine-grained local patterns through forward diffusion and reverse denoising. Finally, we optimize the model in an autoregressive manner. As a result, TimeDART effectively accounts for both global and local sequence features in a coherent way. We conduct extensive experiments on public datasets for time series forecasting and classification. The experimental results demonstrate that TimeDART consistently outperforms previous compared methods, validating the effectiveness of our approach. Our code is available at https://github.com/Melmaphother/TimeDART.

Problem

• captures long-term dynamic evolution
• models fine-grained local patterns
• unifies global and local sequence features

Innovation

• Self-supervised pre-training framework
• Causal Transformer with patch embedding
• Denoising diffusion for local patterns