Diagnosis of Pulmonary Hypertension by Integrating Multimodal Data with a Hybrid Graph Convolutional and Transformer Network

2025-03-28

AI Summary

This study addresses the challenge of early, precise phenotyping in pulmonary hypertension (PH). We propose an end-to-end multimodal deep learning model for three-way classification: non-PH, pre-capillary PH, and post-capillary PH. Our method innovatively integrates graph convolutional networks (GCNs), convolutional neural networks (CNNs), and Transformers into a hybrid architecture, enabling joint modeling—on a graph-structured representation—of dynamic cardiac short-axis and four-chamber cine MRI sequences alongside clinical parameters. Validated against right-heart catheterization (the gold standard), the model achieves an overall AUC of 0.81±0.06 (accuracy=0.73±0.06) on the test set; notably, its AUC for distinguishing pre-capillary PH reaches 0.86±0.06. This marked improvement in PH subphenotype discrimination provides robust, imaging-clinical integrated decision support for personalized therapeutic management.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Early and accurate diagnosis of pulmonary hypertension (PH) is essential for optimal patient management. Differentiating between pre-capillary and post-capillary PH is critical for guiding treatment decisions. This study develops and validates a deep learning-based diagnostic model for PH, designed to classify patients as non-PH, pre-capillary PH, or post-capillary PH. This retrospective study analyzed data from 204 patients (112 with pre-capillary PH, 32 with post-capillary PH, and 60 non-PH controls) at the First Affiliated Hospital of Nanjing Medical University. Diagnoses were confirmed through right heart catheterization. We selected 6 samples from each category for the test set (18 samples, 10%), with the remaining 186 samples used for the training set. This process was repeated 35 times for testing. This paper proposes a deep learning model that combines Graph convolutional networks (GCN), Convolutional neural networks (CNN), and Transformers. The model was developed to process multimodal data, including short-axis (SAX) sequences, four-chamber (4CH) sequences, and clinical parameters. Our model achieved a performance of Area under the receiver operating characteristic curve (AUC) = 0.81 +- 0.06(standard deviation) and Accuracy (ACC) = 0.73 +- 0.06 on the test set. The discriminative abilities were as follows: non-PH subjects (AUC = 0.74 +- 0.11), pre-capillary PH (AUC = 0.86 +- 0.06), and post-capillary PH (AUC = 0.83 +- 0.10). It has the potential to support clinical decision-making by effectively integrating multimodal data to assist physicians in making accurate and timely diagnoses.

Problem

• Diagnosing pulmonary hypertension using multimodal data
• Differentiating pre-capillary and post-capillary PH types
• Developing a hybrid GCN-Transformer model for PH classification

Innovation

• Hybrid GCN and Transformer network
• Multimodal data integration
• Deep learning for PH classification