CDPDNet: Integrating Text Guidance with Hybrid Vision Encoders for Medical Image Segmentation

2025-05-25

AI Summary

Medical image segmentation faces two key challenges: (1) sparse annotations in public datasets hinder cross-dataset anatomical representation learning; and (2) purely vision-based models struggle to capture complex anatomical relationships and task-specific variations, limiting accuracy and generalizability. To address these, we propose a text-guided hybrid vision segmentation framework featuring a novel CLIP-DINO dual-encoder collaboration mechanism and a Text-driven Task Prompt Generation (TTPG) module—enabling cross-dataset anatomical semantic alignment and fine-grained task discrimination. Our method integrates DINOv2 self-supervised ViT and CNN backbones, multi-head cross-modal attention, CLIP text embedding projection, and learnable text prompt modulation. Evaluated across multiple medical imaging benchmarks, it significantly outperforms state-of-the-art methods in segmentation accuracy, cross-dataset generalization, and robustness to partial annotations.

Technology Category

• Computer Vision: Segmentation
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Multimodal Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Security and Privacy: Data transparency and provenance

Abstract

Most publicly available medical segmentation datasets are only partially labeled, with annotations provided for a subset of anatomical structures. When multiple datasets are combined for training, this incomplete annotation poses challenges, as it limits the model's ability to learn shared anatomical representations among datasets. Furthermore, vision-only frameworks often fail to capture complex anatomical relationships and task-specific distinctions, leading to reduced segmentation accuracy and poor generalizability to unseen datasets. In this study, we proposed a novel CLIP-DINO Prompt-Driven Segmentation Network (CDPDNet), which combined a self-supervised vision transformer with CLIP-based text embedding and introduced task-specific text prompts to tackle these challenges. Specifically, the framework was constructed upon a convolutional neural network (CNN) and incorporated DINOv2 to extract both fine-grained and global visual features, which were then fused using a multi-head cross-attention module to overcome the limited long-range modeling capability of CNNs. In addition, CLIP-derived text embeddings were projected into the visual space to help model complex relationships among organs and tumors. To further address the partial label challenge and enhance inter-task discriminative capability, a Text-based Task Prompt Generation (TTPG) module that generated task-specific prompts was designed to guide the segmentation. Extensive experiments on multiple medical imaging datasets demonstrated that CDPDNet consistently outperformed existing state-of-the-art segmentation methods. Code and pretrained model are available at: https://github.com/wujiong-hub/CDPDNet.git.

Problem

• Addresses incomplete annotations in medical image datasets
• Enhances anatomical relationship modeling with text guidance
• Improves segmentation accuracy and cross-dataset generalizability

Innovation

• Combines CLIP text embedding with DINOv2 vision features
• Uses multi-head cross-attention for feature fusion
• Generates task-specific text prompts for segmentation