Contrastive Cross-Modal Learning for Infusing Chest X-ray Knowledge into ECGs

2025-06-24

AI Summary

Electrocardiogram (ECG) signals alone exhibit limited diagnostic capability for conditions such as cardiac enlargement, pleural effusion, and pulmonary edema. Method: We propose CroMoTEX, a cross-modal knowledge transfer framework that enables ECG-only inference without requiring chest X-rays (CXRs) at test time. It employs supervised cross-modal contrastive learning to jointly model ECG temporal representations and CXR-derived diagnostic semantics, augmented by an adaptive hard negative weighting mechanism to improve inter-modal alignment. Contribution/Results: Trained end-to-end on the MIMIC-IV-ECG and MIMIC-CXR datasets, CroMoTEX achieves an AUROC of 78.31% for pulmonary edema detection—significantly outperforming state-of-the-art unimodal and cross-modal baselines. This demonstrates the feasibility and clinical relevance of transferring diagnostic knowledge from medical imaging to physiological signal analysis.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Modern diagnostic workflows are increasingly multimodal, integrating diverse data sources such as medical images, structured records, and physiological time series. Among these, electrocardiograms (ECGs) and chest X-rays (CXRs) are two of the most widely used modalities for cardiac assessment. While CXRs provide rich diagnostic information, ECGs are more accessible and can support scalable early warning systems. In this work, we propose CroMoTEX, a novel contrastive learning-based framework that leverages chest X-rays during training to learn clinically informative ECG representations for multiple cardiac-related pathologies: cardiomegaly, pleural effusion, and edema. Our method aligns ECG and CXR representations using a novel supervised cross-modal contrastive objective with adaptive hard negative weighting, enabling robust and task-relevant feature learning. At test time, CroMoTEX relies solely on ECG input, allowing scalable deployment in real-world settings where CXRs may be unavailable. Evaluated on the large-scale MIMIC-IV-ECG and MIMIC-CXR datasets, CroMoTEX outperforms baselines across all three pathologies, achieving up to 78.31 AUROC on edema. Our code is available at github.com/vineetpmoorty/cromotex.

Problem

• Infusing chest X-ray knowledge into ECG representations
• Learning clinically informative ECG features for cardiac pathologies
• Enabling ECG-based diagnosis without requiring chest X-rays at test time

Innovation

• Contrastive learning for ECG-CXR knowledge transfer
• Supervised cross-modal contrastive objective with weighting
• ECG-only inference for scalable deployment