Do Time Series Foundation Model Benchmarks Hide Regime-Dependent Failures? Evidence from Traffic Speed Forecasting

2026-06-16

AI Summary

This study addresses a critical limitation in existing time series foundation models for traffic forecasting: their reliance on aggregate evaluation metrics obscures significant performance degradation during pivotal traffic states, particularly transitions between congestion and free-flow conditions. To uncover this issue, the authors introduce a traffic state–stratified evaluation framework, revealing that mean absolute error (MAE) surges to 11 mph during transition states—compared to just 3 mph overall—and that 90% prediction interval coverage can drop as low as 55%. In response, they propose a Bimodal Mixture Augmentation (BMA) post-processing strategy that integrates historical distribution priors. This approach substantially enhances prediction robustness and interval coverage during transition states while preserving overall accuracy.

Abstract

Standard benchmarks evaluate time series foundation models (TSFMs) using aggregate metrics, but these can mask severe failures in critical operating regimes. We introduce regime-stratified evaluation and apply it to three TSFMs on two standard traffic speed benchmarks. Traffic exhibits abrupt regime switching between free-flow and congested states, producing bimodal speed distributions during transitions. When we stratify by traffic regime, both accuracy and prediction-interval coverage degrade sharply during transitions: transition-regime MAE reaches 11 mph (versus 3 mph overall), and empirical coverage of 90% prediction intervals drops as low as 55%. These failures are invisible in aggregate metrics because free-flow observations dominate the sample. A simple historical conditional baseline (sampling from per-sensor training distributions) achieves better transition coverage than any TSFM, but has far worse overall accuracy. We propose bimodal mixture augmentation (BMA), a post-hoc method that combines TSFM forecasts with historical distributional knowledge, approaching the historical baseline's transition coverage while preserving the TSFM's accuracy. Our results suggest that TSFM benchmarks should incorporate regime-aware evaluation to surface failures that aggregate metrics hide.

Problem

• time series foundation models
• regime-dependent failures
• traffic speed forecasting
• benchmark evaluation
• prediction intervals

Innovation

• regime-stratified evaluation
• time series foundation models
• bimodal mixture augmentation
• prediction interval coverage
• traffic speed forecasting