Towards Interpretable Time Series Foundation Models

2025-07-10

AI Summary

Lightweight instruction-tuned language models lack interpretable temporal reasoning capabilities due to insufficient understanding of time-series semantics. Method: We propose a “linguified knowledge distillation” paradigm: leveraging a large multimodal model (LMM) to automatically generate natural-language annotations—covering trend direction, noise intensity, extremum localization, etc.—for synthetically generated mean-reversion time series, and using these annotations to supervise fine-tuning of a compact Qwen model. Contributions/Results: (1) We introduce the first fine-grained time-series language understanding benchmark explicitly designed for interpretability evaluation; (2) we enable synthetic-data-driven distillation without reliance on ground-truth labels. Experiments demonstrate that the distilled small model significantly outperforms baselines across multiple time-series explanation tasks, while offering advantages for edge deployment and privacy preservation.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Knowledge Representation and Reasoning: Knowledge Representation Languages

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

In this paper, we investigate the distillation of time series reasoning capabilities into small, instruction-tuned language models as a step toward building interpretable time series foundation models. Leveraging a synthetic dataset of mean-reverting time series with systematically varied trends and noise levels, we generate natural language annotations using a large multimodal model and use these to supervise the fine-tuning of compact Qwen models. We introduce evaluation metrics that assess the quality of the distilled reasoning - focusing on trend direction, noise intensity, and extremum localization - and show that the post-trained models acquire meaningful interpretive capabilities. Our results highlight the feasibility of compressing time series understanding into lightweight, language-capable models suitable for on-device or privacy-sensitive deployment. This work contributes a concrete foundation toward developing small, interpretable models that explain temporal patterns in natural language.

Problem

• Develop interpretable time series foundation models using small language models
• Distill time series reasoning capabilities into lightweight, instruction-tuned models
• Enable natural language explanations of temporal patterns for on-device deployment

Innovation

• Distill time series reasoning into small language models
• Use synthetic data and large models for supervision
• Evaluate trend, noise, and extremum interpretation capabilities