TimeDiT: General-purpose Diffusion Transformers for Time Series Foundation Model

2024-09-03

arXiv.org

AI Summary

This work addresses key challenges in time-series modeling—including missing data imputation, multi-resolution analysis, uncertainty quantification, and integration of physical constraints—by proposing TimeDiT, the first general-purpose foundation model for time series. Methodologically, TimeDiT unifies Transformer-based representation learning with diffusion-based generative modeling, featuring a theory-driven unified masking schedule and knowledge-aware denoising process that enables consistent zero-shot and fine-tuned inference across diverse tasks. It introduces the first provably sound, fine-tuning-free model editing strategy, allowing dynamic injection of external domain knowledge during sampling. Extensive experiments demonstrate that TimeDiT achieves state-of-the-art performance across five core tasks: forecasting, imputation, multi-resolution modeling, anomaly detection, and generation. This is the first empirical validation of a prototype foundation model for time series, establishing its feasibility and broad generalization capability.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Computer Vision: Diffusion Models for Vision
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Foundation models, particularly Large Language Models (LLMs), have revolutionized text and video processing, yet time series data presents distinct challenges for such approaches due to domain-specific features such as missing values, multi-resolution characteristics, etc. Furthermore, the de-facto autoregressive transformers tend to learn deterministic temporal dependencies within pre-trained data while overlooking inherent uncertainties and lacking integration of physical constraints. In this paper, we introduce TimeDiT, a diffusion transformer model that synergistically combines transformer-based temporal dependency learning with diffusion-based probabilistic sampling. TimeDiT employs a unified masking mechanism to harmonize the training and inference process across diverse tasks while introducing a theoretically grounded, finetuning-free model editing strategy that enables flexible integration of external knowledge during sampling. Acknowledging the challenges of unifying multiple downstream tasks under a single model, our systematic evaluation demonstrates TimeDiT's effectiveness both in fundamental tasks, i.e., forecasting and imputation, through zero-shot/fine-tuning; and in domain tasks, i.e., multi-resolution forecasting, anomaly detection, and data generation, establishing it as a extit{proto-foundation model} that bridges the gap between general-purpose and domain-specific models.

Problem

• Time series data challenges
• Autoregressive transformers limitations
• Unified model for diverse tasks

Innovation

• Diffusion transformer model
• Unified masking mechanism
• Finetuning-free model editing