ITFormer: Bridging Time Series and Natural Language for Multi-Modal QA with Large-Scale Multitask Dataset

2025-06-24

AI Summary

To address the challenges of integrating high-dimensional time series with natural language and enabling effective interactive understanding, this paper introduces the novel task of Time Series Question Answering (TSQA) and releases EngineMT-QA—the first large-scale, multi-task temporal-textual QA dataset. We propose ITFormer, a lightweight framework that synergizes a frozen large language model with a trainable time-series encoder, incorporating cross-modal feature alignment and dynamic fusion mechanisms to achieve precise multimodal interaction with minimal trainable parameters. Evaluated on a comprehensive multi-scenario TSQA benchmark, ITFormer significantly outperforms strong baselines, achieving an average accuracy gain of +8.2%. The approach delivers high performance, low computational overhead, and model efficiency, establishing a new paradigm for dynamic decision-making tasks in industrial monitoring, clinical diagnosis, and related domains.

Technology Category

• Natural Language Processing: Question Answering
• Machine Learning: Time-Series/Data Streams
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Time-series data are critical in diverse applications, such as industrial monitoring, medical diagnostics, and climate research. However, effectively integrating these high-dimensional temporal signals with natural language for dynamic, interactive tasks remains a significant challenge. To address this, we introduce the Time-Series Question Answering (Time-Series QA) task and release EngineMT-QA, the first large-scale, multi-task, temporal-textual QA dataset designed to capture complex interactions between time-series signals and natural language. Building on this resource, we propose the Instruct Time Transformer (ITFormer), a novel framework that bridges time-series encoders with frozen large language models (LLMs). ITFormer effectively extracts, aligns, and fuses temporal and textual features, achieving a strong improvement in QA accuracy over strong baselines with fewer than 1% additional trainable parameters. By combining computational efficiency with robust cross-modal modeling, our work establishes a adaptable paradigm for integrating temporal data with natural language, paving the way for new research and applications in multi-modal AI. More details about the project, including datasets and code, are available at: https://pandalin98.github.io/itformer_site/

Problem

• Integrating time-series data with natural language for interactive tasks
• Creating large-scale multi-task dataset for temporal-textual QA
• Developing efficient framework to align temporal and textual features

Innovation

• Introduces large-scale multi-task temporal-textual QA dataset
• Proposes ITFormer bridging time-series encoders with LLMs
• Achieves high QA accuracy with minimal trainable parameters