Understanding the Implicit Biases of Design Choices for Time Series Foundation Models

2025-10-22

AI Summary

This work systematically investigates implicit biases introduced by design choices—such as patch size, embedding strategy, and pretraining objectives—in time series foundation models (TSFMs), and their impact on fundamental model properties including temporal dynamics, geometric structure, and mean-reversion tendency. Employing a combined approach of theoretical modeling and controlled empirical analysis, the study uncovers how distinct architectural and objective-level decisions induce both intuitive and counterintuitive behavioral patterns. It further characterizes, for the first time, the nonlinear interplay among multiple bias types in downstream tasks such as anomaly detection. Key findings reveal that specific design choices substantially distort TSFMs’ temporal inductive biases and statistical robustness. The work establishes an interpretable bias diagnostic framework and provides principled guidelines for enhancing robustness, thereby advancing TSFM development from “black-box fitting” toward a mechanism-driven, scientifically grounded paradigm.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Cognitive Modeling & Cognitive Systems: Other Foundations of Cognitive Modeling & Systems
• Computer Vision: Other Foundations of Computer Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

Time series foundation models (TSFMs) are a class of potentially powerful, general-purpose tools for time series forecasting and related temporal tasks, but their behavior is strongly shaped by subtle inductive biases in their design. Rather than developing a new model and claiming that it is better than existing TSFMs, e.g., by winning on existing well-established benchmarks, our objective is to understand how the various ``knobs'' of the training process affect model quality. Using a mix of theory and controlled empirical evaluation, we identify several design choices (patch size, embedding choice, training objective, etc.) and show how they lead to implicit biases in fundamental model properties (temporal behavior, geometric structure, how aggressively or not the model regresses to the mean, etc.); and we show how these biases can be intuitive or very counterintuitive, depending on properties of the model and data. We also illustrate in a case study on outlier handling how multiple biases can interact in complex ways; and we discuss implications of our results for learning the bitter lesson and building TSFMs.

Problem

• Investigating how training design choices create implicit biases in time series foundation models
• Analyzing how patch size and embedding selection affect temporal behavior and geometric structure
• Understanding complex interactions between multiple biases in time series model performance

Innovation

• Analyzing design choices' implicit biases
• Using theory and controlled empirical evaluation
• Identifying biases in model properties