🤖 AI Summary
Financial time series exhibit non-stationarity and regime-switching behavior, which challenge conventional Transformers due to their implicit stationarity assumptions. To address this limitation, this work proposes the TIPS framework, which dynamically integrates multiple inductive biases—causality, locality, and periodicity—into the Transformer architecture for the first time. Specifically, bias-specific teacher models generate attention masks, and a regime-aware knowledge distillation strategy is devised to enable robust forecasting in non-stationary markets. Empirical evaluations demonstrate that TIPS outperforms strong ensemble baselines by 55% in annualized returns, 9% in Sharpe ratio, and 16% in Calmar ratio across four major stock markets, while requiring only 38% of the baseline’s inference computation.
📝 Abstract
Transformer-based models have been widely adopted for time-series forecasting due to their high representational capacity and architectural flexibility. However, many Transformer variants implicitly assume stationarity and stable temporal dynamics -- assumptions routinely violated in financial markets characterized by regime shifts and non-stationarity. Empirically, state-of-the-art time-series Transformers often underperform even vanilla Transformers on financial tasks, while simpler architectures with distinct inductive biases, such as CNNs and RNNs, can achieve stronger performance with substantially lower complexity. At the same time, no single inductive bias dominates across markets or regimes, suggesting that robust financial forecasting requires integrating complementary temporal priors. We propose TIPS (Transformer with Inductive Prior Synthesis), a knowledge distillation framework that synthesizes diverse inductive biases -- causality, locality, and periodicity -- within a unified Transformer. TIPS trains bias-specialized Transformer teachers via attention masking, then distills their knowledge into a single student model with regime-dependent alignment across inductive biases. Across four major equity markets, TIPS achieves state-of-the-art performance, outperforming strong ensemble baselines by 55%, 9%, and 16% in annual return, Sharpe ratio, and Calmar ratio, while requiring only 38% of the inference-time computation. Further analyses show that TIPS generates statistically significant excess returns beyond both vanilla Transformers and its teacher ensembles, and exhibits regime-dependent behavioral alignment with classical architectures during their profitable periods. These results highlight the importance of regime-dependent inductive bias utilization for robust generalization in non-stationary financial time series.