UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHR

2026-01-25

AI Summary

This work addresses the challenge of integrating heterogeneous clinical data—such as structured electronic health records (EHR) and raw electrocardiogram (ECG) signals—for accurate prognosis prediction, a task hindered by the inability of large language models (LLMs) to directly process non-textual modalities. To overcome this limitation, the authors propose UniPACT, a novel framework that introduces a structured prompting mechanism to convert numerical EHR entries into semantically meaningful text, which is then unified with encoded raw ECG waveforms as input to an LLM for end-to-end multimodal prognostic question answering. The approach enables joint multitask inference, achieving an average AUROC of 89.37% on the MDS-ED benchmark—significantly outperforming specialized baseline models—and demonstrates robustness under missing data conditions.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Accurate clinical prognosis requires synthesizing structured Electronic Health Records (EHRs) with real-time physiological signals like the Electrocardiogram (ECG). Large Language Models (LLMs) offer a powerful reasoning engine for this task but struggle to natively process these heterogeneous, non-textual data types. To address this, we propose UniPACT (Unified Prognostic Question Answering for Clinical Time-series), a unified framework for prognostic question answering that bridges this modality gap. UniPACT's core contribution is a structured prompting mechanism that converts numerical EHR data into semantically rich text. This textualized patient context is then fused with representations learned directly from raw ECG waveforms, enabling an LLM to reason over both modalities holistically. We evaluate UniPACT on the comprehensive MDS-ED benchmark, it achieves a state-of-the-art mean AUROC of 89.37% across a diverse set of prognostic tasks including diagnosis, deterioration, ICU admission, and mortality, outperforming specialized baselines. Further analysis demonstrates that our multimodal, multi-task approach is critical for performance and provides robustness in missing data scenarios.

Problem

• prognostic question answering
• multimodal integration
• ECG
• Electronic Health Records
• clinical prognosis

Innovation

• multimodal learning
• structured prompting
• ECG representation
• prognostic question answering
• LLM for healthcare