On the Internal Semantics of Time-Series Foundation Models

2025-11-19

AI Summary

This work investigates how time-series foundation models (TSFMs) internally represent fundamental temporal concepts. Addressing four key problems—hierarchical concept encoding, linear recoverability of atomic concepts, depth-wise evolution of representations, and compositional concept handling—we propose a systematic probing framework integrating layer-wise linear probes, representation similarity analysis, and concept disentanglement metrics. We find that TSFMs exhibit a clear semantic hierarchy: shallow layers capture local temporal patterns, while deeper layers encode discrete events and change-point signals. Atomic concepts are reliably localized, yet spectral and deformation factors prove most challenging to recover linearly. Probe performance degrades significantly for composite concepts, revealing representational interference as a bottleneck in modeling interactive dynamics. This study provides the first empirical characterization of temporal semantic hierarchies in TSFMs, offering theoretical foundations for interpretable modeling and architectural refinement.

Technology Category

• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Time-series Foundation Models (TSFMs) have recently emerged as a universal paradigm for learning across diverse temporal domains. However, despite their empirical success, the internal mechanisms by which these models represent fundamental time-series concepts remain poorly understood. In this work, we undertake a systematic investigation of concept interpretability in TSFMs. Specifically, we examine: (i) which layers encode which concepts, (ii) whether concept parameters are linearly recoverable, (iii) how representations evolve in terms of concept disentanglement and abstraction across model depth, and (iv) how models process compositions of concepts. We systematically probe these questions using layer-wise analyses, linear recoverability tests, and representation similarity measures, providing a structured account of TSFM semantics. The resulting insights show that early layers mainly capture local, time-domain patterns (e.g., AR(1), level shifts, trends), while deeper layers encode dispersion and change-time signals, with spectral and warping factors remaining the hardest to recover linearly. In compositional settings, however, probe performance degrades, revealing interference between concepts. This highlights that while atomic concepts are reliably localized, composition remains a challenge, underscoring a key limitation in current TSFMs'ability to represent interacting temporal phenomena.

Problem

• Investigating how time-series foundation models internally represent fundamental temporal concepts
• Analyzing layer-wise encoding, linear recoverability, and concept evolution across model depth
• Examining model limitations in handling compositions of interacting temporal concepts

Innovation

• Layer-wise analysis of time-series concept encoding
• Linear recoverability tests for concept parameters
• Representation similarity measures across model depth