Self-adaptive vision-language model for 3D segmentation of pulmonary artery and vein

2025-01-07

AI Summary

To address the challenges of scarce annotated samples and insufficient cross-modal representation fusion in 3D CT segmentation of pulmonary arteries/veins, this paper proposes a language-guided adaptive cross-attention segmentation framework. Methodologically: (1) CLIP’s pre-trained text-image joint semantic features are leveraged; (2) learnable adapters enable efficient fine-tuning of CLIP on sparsely labeled 3D medical images; (3) an adaptive cross-attention mechanism dynamically fuses multi-modal representations and is embedded into the 3D U-Net decoder. Evaluated on the largest publicly available pulmonary artery/vein CT dataset to date (718 cases), our method significantly outperforms state-of-the-art approaches while reducing annotation requirements by over 60%. The code and dataset will be made publicly available.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Accurate segmentation of pulmonary structures iscrucial in clinical diagnosis, disease study, and treatment planning. Significant progress has been made in deep learning-based segmentation techniques, but most require much labeled data for training. Consequently, developing precise segmentation methods that demand fewer labeled datasets is paramount in medical image analysis. The emergence of pre-trained vision-language foundation models, such as CLIP, recently opened the door for universal computer vision tasks. Exploiting the generalization ability of these pre-trained foundation models on downstream tasks, such as segmentation, leads to unexpected performance with a relatively small amount of labeled data. However, exploring these models for pulmonary artery-vein segmentation is still limited. This paper proposes a novel framework called Language-guided self-adaptive Cross-Attention Fusion Framework. Our method adopts pre-trained CLIP as a strong feature extractor for generating the segmentation of 3D CT scans, while adaptively aggregating the cross-modality of text and image representations. We propose a s pecially designed adapter module to fine-tune pre-trained CLIP with a self-adaptive learning strategy to effectively fuse the two modalities of embeddings. We extensively validate our method on a local dataset, which is the largest pulmonary artery-vein CT dataset to date and consists of 718 labeled data in total. The experiments show that our method outperformed other state-of-the-art methods by a large margin. Our data and code will be made publicly available upon acceptance.

Problem

• 3D Image Segmentation
• Pulmonary Artery-Vein Separation
• Deep Learning Limitations

Innovation

• Language-Guided Adaptive Cross-Attention Fusion
• CLIP Model
• 3D Lung Vasculature Segmentation