Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography

2026-06-24

AI Summary

This work addresses the limitations of existing vision–language pretraining methods on 3D CT imaging, which suffer from inefficient visual backbones and coarse-grained semantic alignment. To overcome these challenges, the authors propose a disease-oriented, fine-grained alignment framework that integrates a hybrid 3D CNN–Vision Transformer encoder and learnable disease query tokens to disentangle lesion representations. Additionally, they introduce a clinical phrase–driven, diagnosis-aware prompting mechanism to enhance semantic grounding. The proposed method achieves AUC scores of 84.4% and 75.4% on CT-RATE and Rad-ChestCT benchmarks, respectively—improving upon baseline models by 5.1% and 5.4%. It further demonstrates a performance gain of up to 9.8% across a 60-disease classification task and significantly boosts zero-shot diagnostic capability and radiology report generation through improved transferability.

Abstract

Vision-language pre-training (VLP) holds great promise for general-purpose medical AI by leveraging radiology reports as rich textual supervision, yet existing methods struggle with 3D CT imaging due to inefficient visual backbones and coarse semantic alignment. To address these issues, we propose a tailored VLP framework featuring three key components: (1) a CNN-ViT hybrid encoder that replaces ViT's patch embedding with a 3D CNN backbone to efficiently capture local anatomical details while preserving global attention and compatibility with pre-trained cross-modal priors; (2) a disease-level contrastive learning mechanism using learnable query tokens to dynamically extract disease-specific semantics from full reports and align them with corresponding visual features, thereby disentangling distinct diseases within the same anatomical region; and (3) a diagnosis-aware prompt strategy that employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap and enhance zero-shot diagnostic reliability. Our model achieves state-of-the-art performance on CT-RATE (84.4% AUC, +5.1%) and Rad-ChestCT (75.4% AUC, +5.4%), with even larger gains (+9.8% AUC) on a challenging 60-disease benchmark, and demonstrates strong transferability to radiology report generation, underscoring the generality and clinical utility of our approach.

Problem

• Vision-Language Pretraining
• 3D Computed Tomography
• Disease-Centric Alignment
• Semantic Disentanglement
• Medical AI

Innovation

• vision-language pretraining
• 3D CT
• hybrid visual encoding
• disease-level contrastive learning
• diagnosis-aware prompting