Does Text Actually Help? Uncovering and Resolving Text Collapse in Multimodal Time Series Forecasting

2026-06-17

AI Summary

This work addresses the prevalent "text collapse" issue in multimodal time series forecasting, where the textual branch degenerates into a constant mapping independent of the input and fails to provide useful information. The study formally defines this phenomenon for the first time and introduces REST-TS, a novel approach that decouples the numerical backbone from the textual branch and incorporates a residual-specific supervision mechanism. This design explicitly compels the textual branch to model prediction residuals that are difficult for the numerical model to capture. By transforming modality asymmetry into a structural advantage and enabling end-to-end training, REST-TS achieves state-of-the-art performance across multiple real-world datasets and backbone architectures, substantially enhancing both the utilization of textual modalities and their contribution to predictive accuracy.

Abstract

Multimodal time series forecasting, which pairs numerical sequences with domain-relevant textual reports, promises to inject world knowledge into forecasting pipelines. However, we uncover a critical failure mode in existing frameworks that we term text collapse: the text branch converges to a content-independent transformation, contributing negligible discriminative signal regardless of the input description. We argue that text collapse is a consequence of a fundamental asymmetry in time series forecasting: the numerical input is strongly autocorrelated with the output, making the numerical backbone inherently dominant, while the text branch, despite carrying complementary and often critical information, is insufficiently utilized, leading to its systematic underexploitation. To address this, we propose \textbf{REST-TS} (\textbf{R}esidual-\textbf{E}xclusive \textbf{S}upervision for \textbf{T}ext in \textbf{T}ime \textbf{S}eries), which turns the asymmetry into a design principle: the numerical backbone produces its own independent numerical forecast, and the text branch is exclusively supervised to predict the structured components of the residual, the prediction gap that numbers cannot explain. Because no numerical pathway can reduce these losses, the text branch must extract genuine content from the input description. Evaluated across diverse real-world domains and backbone architectures, REST-TS achieves state-of-the-art performance and consistently demonstrates greater text-branch utilization than existing frameworks, providing strong empirical evidence that supervising the text branch on the residual compels it to extract genuine content from the input.

Problem

• multimodal time series forecasting
• text collapse
• text utilization
• forecasting asymmetry
• residual prediction

Innovation

• text collapse
• multimodal time series forecasting
• residual-exclusive supervision
• REST-TS
• asymmetry-aware learning