Semantics or Structure? Auditing Text Sensitivity in Multimodal Time-Series Forecasting

2026-08-23

AI Summary

研究通过文本扰动和归因分析，探讨了多模态时间序列预测中模型是否对文本语义敏感，发现文本内容并非提升预测性能的关键因素。

Abstract

Multimodal time-series forecasting has emerged as a promising paradigm in which natural-language context is expected to improve predictive performance. Recent multimodal foundation models, including Aurora, as well as early- and late-fusion approaches such as MM-TSFlib and TaTS, report substantial gains over unimodal baselines on the Time-MMD benchmark, attributing these improvements to textual information. However, whether these models are actually sensitive to the semantic content of the text remains unverified. We address this question through controlled text perturbations, attribution analyses, and probes of Aurora's text pathway. On Time-MMD, swapping each row's text for any other real text (empty, constant, within-domain shuffled, or cross-domain) moves mean MSE by less than $0.5\%$ on all three architectures. The improvement reported in the literature is recovered when a co-shipped numeric column is removed without touching text. We conclude that, on this benchmark and within this family of frozen-encoder architectures, text content is not the operative signal behind the reported gains. To support future work on text integration in multimodal foundation models for structured data, we release our perturbation protocol and evaluation harness as a reusable diagnostic toolkit.

Problem

• Multimodal time-series forecasting
• Text sensitivity
• Semantic content
• Time-MMD benchmark

Innovation

• text perturbation
• attribution analysis
• multimodal time-series forecasting
• semantic content