TS-Fault: Benchmarking Time Series Forecasters Against Structural Faults

2026-06-16

AI Summary

This work addresses the limited reliability assessment of existing time series forecasting models, which are typically evaluated only on clean data using average error metrics and thus fail to reflect performance under realistic structured corruptions. Treating robustness as a data quality issue, the study systematically defines four categories of structured fault patterns and introduces TS-Fault, a new benchmark that injects semantically controllable faults into critical prediction windows via a unified importance scoring mechanism. The framework employs an orthogonal design across observation/mechanism layers and univariate/multivariate settings, paired with a clean–corrupted data evaluation protocol. Evaluations across 21 models reveal a negative correlation between accuracy on clean data and robustness; notably, mechanism-level faults not only substantially reorder model rankings but also induce widespread catastrophic failures, with foundation models exhibiting the greatest vulnerability.

Abstract

Time series forecasting (TSF) underpins consequential decisions in energy, transportation, finance, and healthcare, yet TSF models are almost universally ranked by a single number (e.g., average error) on clean held-out data, under the implicit assumption that it predicts deployed reliability. However, real faults are not i.i.d noise but structured events with temporal shape, broken cross-variable dependencies, regime change coupled with missingness, and causal propagation across a sensing pipeline. Treating TSF robustness as a data-quality problem, we present TS-Fault, a benchmark that evaluates forecasting models under explicit, parameterized fault scenarios with controllable semantic difficulty. TS-Fault organizes recurring failures into four modes along two orthogonal axes (observation- vs mechanism-level; univariate vs multivariate) and injects each fault into the most prediction-critical window via a unified importance score. This design enables robustness to be tested against the structures models actually rely on, rather than reduced to generic noise sensitivity. We evaluate 21 models across 6 datasets, 4 modes, and 5 difficulty levels under a paired clean/corrupt protocol. The results reveal three findings that contradict common leaderboard intuition: (i) clean-data accuracy anti-correlates with robustness; (ii) clean rankings are preserved under observation-level faults but reshuffled under mechanism-level faults; and (iii) all catastrophic failures occur under mechanism-level faults, with foundation models achieving the highest clean-data accuracy yet exhibiting the greatest fragility. The code is publicly available at https://github.com/Ray-zyy/TS-Fault.

Problem

• time series forecasting
• structural faults
• model robustness
• fault scenarios
• reliability evaluation

Innovation

• time series forecasting
• robustness benchmark
• structured faults
• mechanism-level faults
• fault injection