OpenTSLM: Time-Series Language Models for Reasoning over Multivariate Medical Text- and Time-Series Data

2025-10-02

AI Summary

Existing large language models (LLMs) struggle to effectively model multivariate medical time series, limiting their utility in clinical decision support. To address this, we propose OpenTSLM—the first LLM architecture that natively incorporates time series as a first-class modality during pretraining. It employs soft prompt initialization and Flamingo-style cross-modal attention to enable joint reasoning over textual and long-horizon sequential data. The method supports memory-efficient training, chain-of-thought reasoning, and seamless multimodal fusion. Evaluated on sleep staging (F1 = 69.9) and human activity recognition (accuracy = 65.4%), OpenTSLM significantly outperforms established baselines. Notably, its compact-parameter variant surpasses GPT-4o in domain-specific performance and has been validated by clinical experts for robust medical reasoning. This work establishes a scalable, high-fidelity paradigm for temporal understanding in multimodal medical AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

LLMs have emerged as powerful tools for interpreting multimodal data. In medicine, they hold particular promise for synthesizing large volumes of clinical information into actionable insights and digital health applications. Yet, a major limitation remains their inability to handle time series. To overcome this gap, we present OpenTSLM, a family of Time Series Language Models (TSLMs) created by integrating time series as a native modality to pretrained LLMs, enabling reasoning over multiple time series of any length. We investigate two architectures for OpenTSLM. The first, OpenTSLM-SoftPrompt, models time series implicitly by concatenating learnable time series tokens with text tokens via soft prompting. Although parameter-efficient, we hypothesize that explicit time series modeling scales better and outperforms implicit approaches. We thus introduce OpenTSLM-Flamingo, which integrates time series with text via cross-attention. We benchmark both variants against baselines that treat time series as text tokens or plots, across a suite of text-time-series Chain-of-Thought (CoT) reasoning tasks. We introduce three datasets: HAR-CoT, Sleep-CoT, and ECG-QA-CoT. Across all, OpenTSLM models outperform baselines, reaching 69.9 F1 in sleep staging and 65.4 in HAR, compared to 9.05 and 52.2 for finetuned text-only models. Notably, even 1B-parameter OpenTSLM models surpass GPT-4o (15.47 and 2.95). OpenTSLM-Flamingo matches OpenTSLM-SoftPrompt in performance and outperforms on longer sequences, while maintaining stable memory requirements. By contrast, SoftPrompt grows exponentially in memory with sequence length, requiring around 110 GB compared to 40 GB VRAM when training on ECG-QA with LLaMA-3B. Expert reviews by clinicians find strong reasoning capabilities exhibited by OpenTSLMs on ECG-QA. To facilitate further research, we provide all code, datasets, and models open-source.

Problem

• Enabling reasoning over multivariate medical time-series data with language models
• Overcoming LLM limitations in handling time series data of varying lengths
• Integrating time series as native modality for clinical Chain-of-Thought reasoning

Innovation

• Integrates time series as native modality to LLMs
• Uses soft prompting for implicit time series modeling
• Employs cross-attention for explicit time series integration