A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysis

2025-08-18

AI Summary

Cardiovascular multimodal analysis faces challenges including data scarcity, rigid modality configurations, neglect of complementary information during cross-modal alignment, and limitations of single-task learning. To address these, we propose MedFlexFusion—a novel framework enabling dynamic fusion of laboratory tests, electrocardiograms (ECGs), and echocardiograms for the first time. It incorporates a text-guided, task-adaptive representation module to jointly model modality-specificity and complementarity; introduces a cross-modal dynamic alignment strategy that supersedes conventional similarity-driven approaches; and supports multi-task collaborative analysis—including diagnosis, risk stratification, and medical information retrieval. The method integrates advances in biomedical signal processing, vision modeling, and multimodal deep learning. Extensive experiments demonstrate significant improvements over state-of-the-art methods across multiple clinical tasks, with robustness and generalizability validated on independent public benchmarks.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Contemporary cardiovascular management involves complex consideration and integration of multimodal cardiac datasets, where each modality provides distinct but complementary physiological characteristics. While the effective integration of multiple modalities could yield a holistic clinical profile that accurately models the true clinical situation with respect to data modalities and their relatives weightings, current methodologies remain limited by: 1) the scarcity of patient- and time-aligned multimodal data; 2) reliance on isolated single-modality or rigid multimodal input combinations; 3) alignment strategies that prioritize cross-modal similarity over complementarity; and 4) a narrow single-task focus. In response to these limitations, a comprehensive multimodal dataset was curated for immediate application, integrating laboratory test results, electrocardiograms, and echocardiograms with clinical outcomes. Subsequently, a unified framework, Textual Guidance Multimodal fusion for Multiple cardiac tasks (TGMM), was proposed. TGMM incorporated three key components: 1) a MedFlexFusion module designed to capture the unique and complementary characteristics of medical modalities and dynamically integrate data from diverse cardiac sources and their combinations; 2) a textual guidance module to derive task-relevant representations tailored to diverse clinical objectives, including heart disease diagnosis, risk stratification and information retrieval; and 3) a response module to produce final decisions for all these tasks. Furthermore, this study systematically explored key features across multiple modalities and elucidated their synergistic contributions in clinical decision-making. Extensive experiments showed that TGMM outperformed state-of-the-art methods across multiple clinical tasks, with additional validation confirming its robustness on another public dataset.

Problem

• Integrating multimodal cardiac data for holistic clinical analysis
• Overcoming limitations of single-modality and rigid multimodal approaches
• Enhancing cardiac task performance via dynamic multimodal fusion

Innovation

• MedFlexFusion module integrates diverse cardiac data dynamically
• Textual guidance tailors representations for clinical tasks
• Unified framework TGMM excels in multiple clinical tasks