Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification

2024-01-15

arXiv.org

AI Summary

This study addresses hypertension—a continuous pathological process—by proposing the first ordinal classification-based multimodal representation learning framework for fine-grained, continuous stratification of cardiac functional impairment. The method jointly integrates echocardiographic features (e.g., ejection fraction, strain) with structured clinical records. It employs modality-specific feature projection coupled with an XTab Transformer encoder and incorporates an ordinal classification loss to explicitly model the pathological progression spectrum. Evaluated on a cohort of 239 hypertensive patients, the model achieves an AUROC of 98%, demonstrates high reproducibility in stratification (MAE < 3.6%), and uncovers physiologically plausible dynamic patterns of functional deterioration. Moreover, it identifies several novel potential imaging–clinical biomarkers. This work advances precision phenotyping in hypertension by bridging continuous pathophysiology with interpretable, ordinal-aware multimodal learning.

Technology Category

• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Deep learning enables automatic and robust extraction of cardiac function descriptors from echocardiographic sequences, such as ejection fraction or strain. These descriptors provide fine-grained information that physicians consider, in conjunction with more global variables from the clinical record, to assess patients' condition. Drawing on novel transformer models applied to tabular data, we propose a method that considers all descriptors extracted from medical records and echocardiograms to learn the representation of a cardiovascular pathology with a difficult-to-characterize continuum, namely hypertension. Our method first projects each variable into its own representation space using modality-specific approaches. These standardized representations of multimodal data are then fed to a transformer encoder, which learns to merge them into a comprehensive representation of the patient through the task of predicting a clinical rating. This stratification task is formulated as an ordinal classification to enforce a pathological continuum in the representation space. We observe the major trends along this continuum on a cohort of 239 hypertensive patients, providing unprecedented details in the description of hypertension's impact on various cardiac function descriptors. Our analysis shows that i) the XTab foundation model's architecture allows to reach outstanding performance (98% AUROC) even with limited data (less than 200 training samples), ii) stratification across the population is reproducible between trainings (within 3.6% MAE), and iii) patterns emerge in descriptors, some of which align with established physiological knowledge about hypertension, while others could pave the way for a more comprehensive understanding of this pathology.

Problem

• Fusing echocardiography images and medical records for patient stratification
• Learning representation of hypertension's pathological continuum using Transformers
• Predicting clinical ratings through multimodal data integration

Innovation

• Transformer model fuses echocardiography and medical records
• Modality-specific representations merged via Transformer encoder
• Ordinal classification enforces pathological continuum representation