Rethinking Multimodal Time-Series Forecasting Evaluation

2026-07-07

AI Summary

Existing time series forecasting benchmarks often suffer from limited data scale, synthetic characteristics, homogeneous textual contexts, and data leakage during evaluation, which collectively hinder accurate assessment of model generalization. To address these limitations, this work proposes TimesX—the first multimodal time series forecasting benchmark that simultaneously ensures realism, diversity, and leakage-free evaluation. TimesX encompasses high-quality real-world time series from multiple domains, each paired with rich and varied textual contexts, and features an automated data generation pipeline to facilitate zero-shot research. Experiments on TimesX reveal that many methods excelling on prior benchmarks exhibit significant performance degradation, whereas a simple yet effective ensemble approach leveraging textual context substantially outperforms strong existing baselines, thereby exposing the shortcomings of current methodologies in more realistic scenarios.

Abstract

We introduce a new context-enriched, multimodal time series forecasting benchmark, TimesX. TimesX contains a wide selection of high-quality real-world time series with diverse domains and textual contexts obtained from an automated data generation pipeline, which helps address three main issues of existing multimodal forecasting benchmarks: (1) poor generalization due to the small scale and synthetic nature of benchmark data, (2) very limited types of textual contexts in the benchmarks, and (3) an inability to mitigate data leakage in evaluation. We conduct a thorough empirical study of zero-shot multimodal forecasting approaches on TimesX. Our results suggest that many approaches that perform well on existing benchmarks may fail on TimesX. In contrast, simple ensemble methods that leverage rich textual context accompanying time-series can outperform strong baselines on TimesX.

Problem

• multimodal time-series forecasting
• benchmark evaluation
• data leakage
• textual context
• generalization

Innovation

• multimodal time-series forecasting
• benchmark
• textual context
• data leakage mitigation
• zero-shot forecasting