Towards Cardiac MRI Foundation Models: Comprehensive Visual-Tabular Representations for Whole-Heart Assessment and Beyond

2025-04-17

AI Summary

Current cardiac health assessment is limited by the disconnection between cardiovascular magnetic resonance (CMR) imaging and patient-level clinical factors—such as demographics, metabolism, and lifestyle—leading to incomplete risk characterization. To address this, we propose ViTa, the first vision-tabular multimodal foundation model tailored for comprehensive cardiac health evaluation. ViTa integrates 3D+T cine MRI (short- and long-axis) with structured clinical variables from 42,000 UK Biobank participants, establishing a unified cross-modal representation framework. It employs a Transformer-based 3D+T dynamic encoder, tabular feature embedding with modality alignment, and combines self-supervised pretraining with multi-task fine-tuning. ViTa achieves state-of-the-art performance across diverse downstream tasks—including myocardial segmentation, ejection fraction estimation, cardiomyopathy classification, and diabetes prediction—while significantly improving personalized disease risk stratification accuracy and clinical interpretability.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Cardiac magnetic resonance imaging is the gold standard for non-invasive cardiac assessment, offering rich spatio-temporal views of the cardiac anatomy and physiology. Patient-level health factors, such as demographics, metabolic, and lifestyle, are known to substantially influence cardiovascular health and disease risk, yet remain uncaptured by CMR alone. To holistically understand cardiac health and to enable the best possible interpretation of an individual's disease risk, CMR and patient-level factors must be jointly exploited within an integrated framework. Recent multi-modal approaches have begun to bridge this gap, yet they often rely on limited spatio-temporal data and focus on isolated clinical tasks, thereby hindering the development of a comprehensive representation for cardiac health evaluation. To overcome these limitations, we introduce ViTa, a step toward foundation models that delivers a comprehensive representation of the heart and a precise interpretation of individual disease risk. Leveraging data from 42,000 UK Biobank participants, ViTa integrates 3D+T cine stacks from short-axis and long-axis views, enabling a complete capture of the cardiac cycle. These imaging data are then fused with detailed tabular patient-level factors, enabling context-aware insights. This multi-modal paradigm supports a wide spectrum of downstream tasks, including cardiac phenotype and physiological feature prediction, segmentation, and classification of cardiac and metabolic diseases within a single unified framework. By learning a shared latent representation that bridges rich imaging features and patient context, ViTa moves beyond traditional, task-specific models toward a universal, patient-specific understanding of cardiac health, highlighting its potential to advance clinical utility and scalability in cardiac analysis.

Problem

• Integrates cardiac MRI and patient data for holistic heart assessment
• Overcomes limited spatio-temporal data in current multi-modal approaches
• Enables unified prediction of cardiac diseases and physiological features

Innovation

• Integrates 3D+T cine stacks from multiple views
• Fuses imaging data with tabular patient factors
• Learns shared latent representation for multi-modal tasks