Bi-level Heterogeneous Learning for Time Series Foundation Models: A Federated Learning Approach

2026-04-08

AI Summary

This work addresses the significant heterogeneity inherent in time series data—both within and across domains—which induces gradient conflicts during mixed-batch training and degrades representation quality. To tackle this challenge, the paper presents the first systematic characterization of time series heterogeneity at dual levels and introduces a fine-grained federated learning approach. This method employs local regularization to construct domain-invariant yet semantically coherent representations, thereby mitigating intra-domain conflicts, and integrates a domain-aware aggregation mechanism to foster effective cross-domain collaboration. The proposed framework enables training high-quality time series foundation models from scratch and consistently outperforms existing centralized and federated baselines across multiple benchmarks, achieving state-of-the-art performance in point forecasting, probabilistic forecasting, and large-scale zero-shot tasks.

Abstract

Heterogeneity in time series data is more pronounced than in vision or language, as temporal dynamics vary substantially across domains and tasks. Existing efforts on training time series foundation models (TSFMs) from scratch are often trained with mixed-batch strategies that merge large-scale datasets, which can cause gradient conflicts and degrade representation quality. To address this, we propose a fine-grained learning method that distills invariant knowledge from heterogeneous series while reducing cross-domain interference. We characterize heterogeneity at two levels: inter-domain and intra-domain. To tackle this bi-level heterogeneity, we design a federated learning method that mitigates intra-domain conflicts by enforcing domain-invariant and semantically consistent representations through local regularization, and addresses inter-domain discrepancies by enhancing cross-domain collaboration via domain-aware aggregation. Experiments across diverse benchmarks show that TSFMs trained with our method consistently outperform both centralized and federated TSFM baselines in point and probabilistic forecasting, while also achieving competitive zero-shot performance at scale, offering a flexible pathway for training TSFMs from scratch in heterogeneous environments.

Problem

• time series foundation models
• heterogeneity
• federated learning
• gradient conflicts
• representation quality

Innovation

• bi-level heterogeneity
• federated learning
• time series foundation models
• domain-invariant representation
• cross-domain collaboration