Into the ORBIT for Time Series: Training Regimes for Foundation Models

2026-08-13

AI Summary

This work addresses the challenges faced by existing time series foundation models when trained on heterogeneous corpora, where issues such as domain imbalance, context dependency, prediction horizon variability, and missing values are difficult to explicitly control. To tackle these limitations, the authors propose ORBIT, a novel training paradigm that employs bootstrap multi-level sampling and omni-range incremental training to explicitly model and regulate the training distribution. ORBIT introduces a three-channel missingness-aware patch encoding scheme and, for the first time, incorporates a Rank-Guided Cross-Depth Alignment mechanism to refine shallow-layer representations without incurring additional inference overhead. Built upon an encoder-only Transformer architecture, ORBIT demonstrates substantial improvements in zero-shot forecasting performance on both GIFT-Eval and fev-bench, effectively generalizing across diverse domains and frequencies.

Abstract

Time series foundation models (TSFMs) have advanced primarily through architectural innovation, while training regimes for large-scale heterogeneous corpora remain under-explored. As a result, pre-training distributions are often poorly controlled with respect to domain imbalance, context requirements, prediction horizons, and missingness. We introduce ORBIT (Omni-Range Bootstrap Incremental Training), a training paradigm that makes this distribution explicit and controllable. ORBIT combines Bootstrap Multi-Level Sampling, which controls dataset exposure and samples records, target variables, context windows, and prediction horizons, with Omni-Range Incremental Training, which varies context lengths and prediction horizons throughout a single training stage. Under ORBIT, we train Falcon-2.0, a simple univariate encoder-only Transformer with missingness-aware triple-channel patch tokenization and parallel patch prediction. We further introduce Rank-Guided Cross-Depth Alignment, a training objective that uses late-layer representations as stop-gradient teachers for shallow layers without additional inference cost. Evaluations on GIFT-Eval and fev-bench demonstrate strong zero-shot forecasting performance across diverse domains and frequencies.

Problem

• time series foundation models
• training regimes
• domain imbalance
• prediction horizons
• missingness

Innovation

• ORBIT
• time series foundation models
• Bootstrap Multi-Level Sampling
• Omni-Range Incremental Training
• Rank-Guided Cross-Depth Alignment