Learning to Unify Deformable Shape and Texture Representations for Cardiac Video Classification

2026-07-08

AI Summary

Existing methods for cardiac video classification typically concatenate deformable shape and texture features in a simplistic manner and treat all cardiac phases uniformly, thereby overlooking the complementary nature of modalities and the varying diagnostic significance across phases. To address this, this work proposes a novel model that jointly learns temporal representations of shape and texture within a unified latent space and introduces a bidirectional cross-attention mechanism to enable adaptive cross-modal fusion grounded in spatiotemporal correspondences, along with phase-aware weighting. Evaluated on cine cardiac magnetic resonance (CMR) video datasets, the proposed approach achieves state-of-the-art performance while enhancing interpretability through attention weights that effectively highlight diagnostically relevant cardiac phases and dominant modalities.

Abstract

Deformable shape representations have proven to be robust complements to texture features in cardiac image classification, offering geometric priors that are invariant to imaging artifacts and intensity variations. However, existing deep networks perform simple concatenation to combine these distinct feature representations, which neither fully exploits their complementary nature nor learns cross-modal feature dependencies. Furthermore, this results in uniform attention across all timepoints; hence ignoring the varying diagnostic importance across the cardiac phases. In this paper, we propose a novel cardiac video classification model that, for the first time, learns temporal features in an integrated space of deformable shape and image texture representations. In particular, we design a bi-directional cross-attention in the latent space to fuse latent deformable shape and image features, allowing each modality to adaptively weight the other based on spatio-temporal correspondence. In contrast to current methods that apply uniform weighting across all the cardiac phases, our approach learns to dynamically adjust the contributions of shape and texture representations, derived from images, over time. We demonstrate state-of-the-art classification performance on a cine cardiac magnetic resonance (CMR) video dataset, achieving improved interpretability from attention mechanisms that identify diagnostically critical cardiac phases and modality contributions.

Problem

• deformable shape
• texture representation
• cardiac video classification
• cross-modal fusion
• temporal attention

Innovation

• deformable shape representation
• cross-attention fusion
• temporal feature learning
• cardiac video classification
• multimodal integration