MM-ISTS: Cooperating Irregularly Sampled Time Series Forecasting with Multimodal Vision-Text LLMs

2026-03-06

AI Summary

This work addresses the challenge that existing methods for irregularly sampled time series (ISTS) forecasting struggle to simultaneously capture contextual semantics and fine-grained temporal dynamics. To this end, we propose MM-ISTS, a novel framework that, for the first time, leverages multimodal large language models (MLLMs) to jointly model visual, textual, and temporal modalities for prediction. The key innovations include a cross-modal image-text encoding module that automatically generates auxiliary images and textual descriptions to enrich temporal understanding, as well as an adaptive query-based feature extractor coupled with a modality-aware alignment mechanism to enable efficient fusion of multi-perspective embeddings. Extensive experiments demonstrate that MM-ISTS significantly improves both prediction accuracy and computational efficiency across multiple real-world datasets.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Irregularly sampled time series (ISTS) are widespread in real-world scenarios, exhibiting asynchronous observations on uneven time intervals across variables. Existing ISTS forecasting methods often solely utilize historical observations to predict future ones while falling short in learning contextual semantics and fine-grained temporal patterns. To address these problems, we achieve MM-ISTS, a multimodal framework augmented by vision-text large language models, that bridges temporal, visual, and textual modalities, facilitating ISTS forecasting. MM-ISTS encompasses a novel two-stage encoding mechanism. In particular, a cross-modal vision-text encoding module is proposed to automatically generate informative visual images and textual data, enabling the capture of intricate temporal patterns and comprehensive contextual understanding, in collaboration with multimodal LLMs (MLLMs). In parallel, ISTS encoding extracts complementary yet enriched temporal features from historical ISTS observations, including multi-view embedding fusion and a temporal-variable encoder. Further, we propose an adaptive query-based feature extractor to compress the learned tokens of MLLMs, filtering out small-scale useful knowledge, which in turn reduces computational costs. In addition, a multimodal alignment module with modality-aware gating is designed to alleviate the modality gap across ISTS, images, and text. Extensive experiments on real data offer insight into the effectiveness of the proposed solutions.

Problem

• Irregularly Sampled Time Series
• Time Series Forecasting
• Contextual Semantics
• Temporal Patterns
• Multimodal Learning

Innovation

• Irregularly Sampled Time Series
• Multimodal LLMs
• Cross-modal Encoding
• Temporal-Variable Encoder
• Modality Alignment