LiveHouse-TS: An Open-world Living Benchmark for Time Series Foundation Models

2026-08-17

AI Summary

为解决静态基准测试无法反映模型在持续变化环境中的表现问题，提出LiveHouse-TS，通过实时数据评估时间序列基础模型的长期有效性。

Abstract

Time Series Foundation Models (TSFMs) have recently emerged as a highly promising paradigm for cross-domain zero-shot forecasting. However, existing evaluation protocols predominantly rely on static benchmarks with fixed historical test windows. While these benchmarks provide a valuable baseline snapshot, they evaluate an average performance on a fixed history, failing to capture how models behave in continuously evolving real-world environments characterized by seasonal variations, distribution shifts, and unexpected events. To bridge this gap, we introduce LiveHouse-TS, the first open-world living benchmark infrastructure for TSFMs. By evaluating models prequentially on real future data in open-world environments, LiveHouse-TS shifts time series benchmarking from snapshot accuracy to continuous temporal validity. Rather than acting as a one-off leaderboard, our infrastructure serves as a continuous time series infrastructure designed to explore vital, long-term scientific questions: Can model rankings be maintained over the long term? Which models remain genuinely robust under distribution shifts? Extensive streaming evaluations across 11 domains with 17 datasets demonstrate that static rankings undergo a dramatic reshuffling under a live protocol.

Problem

• Time Series Foundation Models
• static benchmarks
• real-world environments
• distribution shifts
• unexpected events

Innovation

• Time Series Foundation Models
• Open-world Benchmark
• Continuous Evaluation
• Distribution Shifts