CardioState-JEPA: Delay-Aware Cross-Modal Learning of a Shared Cardiac Representation

2026-08-13

AI Summary

This work addresses the limitation of existing cardiac foundation models, which are confined to single modalities and thus unable to exploit shared physiological information across multimodal signals. The authors propose a physiology-aware Joint Embedding Predictive Architecture (JEPA) that maps electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) into a unified latent space using a shared Transformer encoder, augmented with a learnable delay alignment mechanism to handle inter-modal temporal offsets. The model is pretrained via masked cardiac state prediction in a self-supervised manner—first learning modality-specific structures from large-scale unpaired data, then achieving implicit temporal alignment through limited paired data. Evaluated as a frozen encoder across 25 downstream tasks, it yields substantial performance gains: average AUROC improvements of 8.2 points for PPG classification, 18.8 points for PCG murmur detection, and 15.5 points for ECG classification, with some tasks matching or surpassing methods reliant on clinical text or supervised labels.

Abstract

Electrocardiography (ECG), photoplethysmography (PPG), and phonocardiography (PCG) provide complementary views of the same cardiac cycle, yet existing cardiac foundation models are trained for a single sensing modality, leaving the shared physiology across sensors unexploited. We introduce CardioState-JEPA, a cardiac foundation model to learn a single shared representation jointly across ECG, PPG, and PCG, built on a physiology-aware joint-embedding predictive architecture. The model maps heterogeneous waveforms into a common token space, processes them with a single shared Transformer encoder, and learns by predicting masked latent cardiac states, placing the pretraining target on shared physiology rather than sensor-specific waveform appearance. To handle the temporal offsets between electrical, mechanical, and hemodynamic events, cross-modal prediction uses a learned delay aligner that matches signals at the corresponding cardiac time. Because synchronized multi-sensor recordings are scarce, CardioState-JEPA first learns within-modality structure from abundant unimodal data and then uses paired data to align modalities in latent cardiac time. Evaluated as a frozen encoder across 25 downstream tasks spanning ECG, PPG, and PCG, our encoder improves average PPG classification by 8.2 AUROC points, PCG murmur detection by 18.8 AUROC points, and ECG classification by 15.5 AUROC points over the best self-supervised signal baseline and matches or exceeds cardiac models trained with privileged clinical text or supervised labels on several ECG benchmarks. These results establish that heterogeneous cardiac signals can mutually supervise a single foundation model of cardiac physiology.

Problem

• cardiac foundation model
• cross-modal learning
• shared representation
• temporal alignment
• multi-sensor cardiac signals

Innovation

• joint-embedding predictive architecture
• cross-modal alignment
• learned delay aligner
• shared cardiac representation
• foundation model