HeartFormer: Semantic-Aware Dual-Structure Transformers for 3D Four-Chamber Cardiac Point Cloud Reconstruction

2025-11-28

AI Summary

Traditional cine MRI provides only 2D slices, limiting accurate 3D modeling of cardiac morphology and function. To address this, we propose the first point cloud reconstruction framework for multi-class cardiac structures: (1) a Semantic-Aware Dual-Structure Transformer (SA-DSTNet) generates coarse point clouds; (2) a Semantic-Aware Geometric Feature Refinement Network (SA-GFRTNet) enables progressive geometric refinement; and (3) HeartFormer, the first multi-class point cloud completion network for cardiac anatomy. We further introduce HeartCompv1—a large-scale, publicly available dataset comprising 17,000 high-resolution 3D cardiac meshes and corresponding point clouds. Extensive evaluations on HeartCompv1 and cross-domain experiments on UK Biobank demonstrate that our method significantly outperforms state-of-the-art approaches, achieving high-fidelity, geometrically consistent, and strongly generalizable 3D point cloud reconstruction of the four cardiac chambers.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

We present the first geometric deep learning framework based on point cloud representation for 3D four-chamber cardiac reconstruction from cine MRI data. This work addresses a long-standing limitation in conventional cine MRI, which typically provides only 2D slice images of the heart, thereby restricting a comprehensive understanding of cardiac morphology and physiological mechanisms in both healthy and pathological conditions. To overcome this, we propose extbf{HeartFormer}, a novel point cloud completion network that extends traditional single-class point cloud completion to the multi-class. HeartFormer consists of two key components: a Semantic-Aware Dual-Structure Transformer Network (SA-DSTNet) and a Semantic-Aware Geometry Feature Refinement Transformer Network (SA-GFRTNet). SA-DSTNet generates an initial coarse point cloud with both global geometry features and substructure geometry features. Guided by these semantic-geometry representations, SA-GFRTNet progressively refines the coarse output, effectively leveraging both global and substructure geometric priors to produce high-fidelity and geometrically consistent reconstructions. We further construct extbf{HeartCompv1}, the first publicly available large-scale dataset with 17,000 high-resolution 3D multi-class cardiac meshes and point-clouds, to establish a general benchmark for this emerging research direction. Extensive cross-domain experiments on HeartCompv1 and UK Biobank demonstrate that HeartFormer achieves robust, accurate, and generalizable performance, consistently surpassing state-of-the-art (SOTA) methods. Code and dataset will be released upon acceptance at: https://github.com/10Darren/HeartFormer.

Problem

• Reconstructs 3D heart chambers from 2D MRI scans
• Generates multi-class point clouds for cardiac structures
• Creates a benchmark dataset for cardiac reconstruction research

Innovation

• Dual-structure transformers for multi-class point cloud reconstruction
• Semantic-aware geometry refinement with global and substructure priors
• First large-scale dataset for 3D multi-class cardiac reconstruction