Text Reinforcement for Multimodal Time Series Forecasting

2025-08-30

AI Summary

To address unstable prediction performance in multimodal time series forecasting caused by inaccurate or incomplete historical textual descriptions, this paper proposes Text-enhanced Reinforcement learning (TeR). TeR employs historical time series as supervision signals to dynamically generate semantically enriched text via a differentiable text generator, and incorporates a prediction-feedback-driven reinforcement learning mechanism for end-to-end optimization of text quality. By deeply coupling text generation, multimodal fusion, and policy optimization, TeR enables synergistic enhancement between textual and numerical modalities. Extensive experiments on multiple real-world benchmark datasets demonstrate that TeR consistently outperforms strong baselines and state-of-the-art methods, achieving significant improvements in both forecasting accuracy (MAE/MSE) and cross-scenario stability. These results validate the effectiveness of actively reinforcing the textual modality in multimodal time series modeling.

Technology Category

• Machine Learning: Multimodal Learning
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Recent studies in time series forecasting (TSF) use multimodal inputs, such as text and historical time series data, to predict future values. These studies mainly focus on developing advanced techniques to integrate textual information with time series data to perform the task and achieve promising results. Meanwhile, these approaches rely on high-quality text and time series inputs, whereas in some cases, the text does not accurately or fully capture the information carried by the historical time series, which leads to unstable performance in multimodal TSF. Therefore, it is necessary to enhance the textual content to improve the performance of multimodal TSF. In this paper, we propose improving multimodal TSF by reinforcing the text modalities. We propose a text reinforcement model (TeR) to generate reinforced text that addresses potential weaknesses in the original text, then apply this reinforced text to support the multimodal TSF model's understanding of the time series, improving TSF performance. To guide the TeR toward producing higher-quality reinforced text, we design a reinforcement learning approach that assigns rewards based on the impact of each reinforced text on the performance of the multimodal TSF model and its relevance to the TSF task. We optimize the TeR accordingly, so as to improve the quality of the generated reinforced text and enhance TSF performance. Extensive experiments on a real-world benchmark dataset covering various domains demonstrate the effectiveness of our approach, which outperforms strong baselines and existing studies on the dataset.

Problem

• Enhancing text quality for multimodal time series forecasting
• Addressing text inaccuracies in multimodal forecasting inputs
• Improving forecasting performance through text reinforcement

Innovation

• Text reinforcement model enhances multimodal forecasting
• Reinforcement learning optimizes text quality
• Generated text improves time series understanding