Explainable Multi-modal Time Series Prediction with LLM-in-the-Loop

2025-03-02

AI Summary

Existing time-series forecasting methods often neglect contextual information embedded in auxiliary modalities (e.g., textual reports), leading to suboptimal multimodal fusion and limited interpretability. To address this, we propose TimeXL—a novel LLM-augmented framework featuring a closed-loop “forecast–reflect–refine” collaboration mechanism. TimeXL introduces a prototype-driven temporal encoder and a tripartite LLM architecture comprising specialized roles: Predictor, Reflector, and Refiner. It is the first work to jointly integrate case-based reasoning, text-temporal consistency diagnosis, and feedback-driven dynamic retraining of the encoder within a unified multimodal time-series modeling pipeline. Evaluated on four real-world datasets, TimeXL achieves up to an 8.9% improvement in AUC over state-of-the-art baselines, while generating human-interpretable, multimodal explanations that jointly enhance predictive accuracy and decision trustworthiness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Time series analysis provides essential insights for real-world system dynamics and informs downstream decision-making, yet most existing methods often overlook the rich contextual signals present in auxiliary modalities. To bridge this gap, we introduce TimeXL, a multi-modal prediction framework that integrates a prototype-based time series encoder with three collaborating Large Language Models (LLMs) to deliver more accurate predictions and interpretable explanations. First, a multi-modal prototype-based encoder processes both time series and textual inputs to generate preliminary forecasts alongside case-based rationales. These outputs then feed into a prediction LLM, which refines the forecasts by reasoning over the encoder's predictions and explanations. Next, a reflection LLM compares the predicted values against the ground truth, identifying textual inconsistencies or noise. Guided by this feedback, a refinement LLM iteratively enhances text quality and triggers encoder retraining. This closed-loop workflow -- prediction, critique (reflect), and refinement -- continuously boosts the framework's performance and interpretability. Empirical evaluations on four real-world datasets demonstrate that TimeXL achieves up to 8.9% improvement in AUC and produces human-centric, multi-modal explanations, highlighting the power of LLM-driven reasoning for time series prediction.

Problem

• Integrates multi-modal data for time series prediction
• Enhances prediction accuracy and interpretability using LLMs
• Provides human-centric explanations for time series analysis

Innovation

• Multi-modal prototype-based encoder integrates time series and text.
• Three LLMs collaborate for prediction, critique, and refinement.
• Closed-loop workflow enhances performance and interpretability continuously.