TSPulse: Dual Space Tiny Pre-Trained Models for Rapid Time-Series Analysis

2025-05-19

AI Summary

To address the high parameter count, computational overhead, and deployment constraints of existing time-series pre-trained models, this paper proposes TSPulse—a hyper-compact model with only 1 million parameters. Methodologically, it introduces (i) dual-space masked reconstruction (time-domain + frequency-domain), (ii) disentangled embedding for detail and semantics, (iii) task-aware TSLens fine-tuning, (iv) multi-head trigonometric anomaly fusion, and (v) a hybrid masked zero-shot imputation strategy. Evaluated across four benchmark tasks—UEA time-series classification, TSB-AD anomaly detection, zero-shot imputation, and time-series retrieval—TSPulse achieves improvements of 5–16%, 20%, 50%, and 25%, respectively. It supports CPU-only inference and accelerates pre-training by 10–100× over state-of-the-art methods, significantly enhancing both efficiency and practicality of time-series analysis.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

The rise of time-series pre-trained models has advanced temporal representation learning, but current state-of-the-art models are often large-scale, requiring substantial compute. We introduce TSPulse, ultra-compact time-series pre-trained models with only 1M parameters, specialized to perform strongly across classification, anomaly detection, imputation, and retrieval tasks. TSPulse introduces innovations at both the architecture and task levels. At the architecture level, it employs a dual-space masked reconstruction, learning from both time and frequency domains to capture complementary signals. This is further enhanced by a dual-embedding disentanglement, generating both detailed embeddings for fine-grained analysis and high-level semantic embeddings for broader task understanding. Notably, TSPulse's semantic embeddings are robust to shifts in time, magnitude, and noise, which is important for robust retrieval. At the task level, TSPulse incorporates TSLens, a fine-tuning component enabling task-specific feature attention. It also introduces a multi-head triangulation technique that correlates deviations from multiple prediction heads, enhancing anomaly detection by fusing complementary model outputs. Additionally, a hybrid mask pretraining is proposed to improves zero-shot imputation by reducing pre-training bias. These architecture and task innovations collectively contribute to TSPulse's significant performance gains: 5-16% on the UEA classification benchmarks, +20% on the TSB-AD anomaly detection leaderboard, +50% in zero-shot imputation, and +25% in time-series retrieval. Remarkably, these results are achieved with just 1M parameters, making TSPulse 10-100X smaller than existing pre-trained models. Its efficiency enables GPU-free inference and rapid pre-training, setting a new standard for efficient time-series pre-trained models. Models will be open-sourced soon.

Problem

• Develop ultra-compact time-series models with 1M parameters
• Enhance performance in classification, anomaly detection, imputation, retrieval
• Achieve efficiency with GPU-free inference and rapid pre-training

Innovation

• Dual-space masked reconstruction for time and frequency domains
• Dual-embedding disentanglement for fine and semantic analysis
• Multi-head triangulation technique for enhanced anomaly detection