MoTime: A Dataset Suite for Multimodal Time Series Forecasting

2025-05-21

AI Summary

This paper addresses the emerging challenge of multimodal time series forecasting by introducing the first systematic solution: the Multimodal Time Series Benchmark (MMTSB), encompassing textual, visual, and metadata modalities. It supports both conventional historical-dependency forecasting and zero-shot cold-start forecasting. Methodologically, we propose novel techniques for aligning heterogeneous multimodal data, cross-modal temporal annotation, controllable cold-start dataset partitioning, and a standardized evaluation protocol. Empirical results—first to quantitatively demonstrate significant performance gains from external modalities in short-sequence and cold-start forecasting—reveal principled relationships between modality utility and intrinsic data characteristics. All datasets, code, and analytical results are publicly released to advance time series forecasting toward realistic, complex scenarios.

Technology Category

• Machine Learning: Time-Series/Data Streams
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs

Abstract

While multimodal data sources are increasingly available from real-world forecasting, most existing research remains on unimodal time series. In this work, we present MoTime, a suite of multimodal time series forecasting datasets that pair temporal signals with external modalities such as text, metadata, and images. Covering diverse domains, MoTime supports structured evaluation of modality utility under two scenarios: 1) the common forecasting task, where varying-length history is available, and 2) cold-start forecasting, where no historical data is available. Experiments show that external modalities can improve forecasting performance in both scenarios, with particularly strong benefits for short series in some datasets, though the impact varies depending on data characteristics. By making datasets and findings publicly available, we aim to support more comprehensive and realistic benchmarks in future multimodal time series forecasting research.

Problem

• Addresses lack of multimodal datasets for time series forecasting
• Evaluates modality utility in common and cold-start forecasting scenarios
• Demonstrates external modalities improve forecasting performance variably

Innovation

• Multimodal dataset suite for time series forecasting
• Integrates text, metadata, images with temporal signals
• Supports cold-start and common forecasting scenarios