Language-guided Medical Image Segmentation with Target-informed Multi-level Contrastive Alignments

2024-12-18

arXiv.org

AI Summary

To address the inherent modality gap between language and medical images—leading to insufficient fine-grained cross-modal alignment in language-guided medical image segmentation—this paper proposes a target-aware multi-level contrastive alignment framework. Methodologically: (1) a target-sensitive semantic distance metric is introduced to enable precise matching between disease-relevant local regions and clinical text descriptions; (2) a low-level feature-guided text prompting strategy is designed to strengthen lesion boundary modeling; and (3) a language-driven target enhancement module is incorporated, integrating cross-modal attention with text-conditioned feature modulation to support joint modeling across CT, MRI, and ultrasound modalities. Evaluated on four cross-modal medical datasets, the method achieves an average Dice coefficient improvement of 3.2%, with particularly notable gains in segmenting small lesions and those with ambiguous boundaries.

Technology Category

• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Medical image segmentation is crucial in modern medical image analysis, which can aid into diagnosis of various disease conditions. Recently, language-guided segmentation methods have shown promising results in automating image segmentation where text reports are incorporated as guidance. These text reports, containing image impressions and insights given by clinicians, provides auxiliary guidance. However, these methods neglect the inherent pattern gaps between the two distinct modalities, which leads to sub-optimal image-text feature fusion without proper cross-modality feature alignments. Contrastive alignments are widely used to associate image-text semantics in representation learning; however, it has not been exploited to bridge the pattern gaps in language-guided segmentation that relies on subtle low level image details to represent diseases. Existing contrastive alignment methods typically algin high-level global image semantics without involving low-level, localized target information, and therefore fails to explore fine-grained text guidance for language-guided segmentation. In this study, we propose a language-guided segmentation network with Target-informed Multi-level Contrastive Alignments (TMCA). TMCA enables target-informed cross-modality alignments and fine-grained text guidance to bridge the pattern gaps in language-guided segmentation. Specifically, we introduce: 1) a target-sensitive semantic distance module that enables granular image-text alignment modelling, and 2) a multi-level alignment strategy that directs text guidance on low-level image features. In addition, a language-guided target enhancement module is proposed to leverage the aligned text to redirect attention to focus on critical localized image features. Extensive experiments on 4 image-text datasets, involving 3 medical imaging modalities, demonstrated that our TMCA achieved superior performances.

Problem

• Bridges pattern gaps in language-guided medical image segmentation.
• Enhances cross-modality feature alignment with fine-grained text guidance.
• Improves segmentation by focusing on low-level, localized image details.

Innovation

• Target-informed Multi-level Contrastive Alignments (TMCA)
• Target-sensitive semantic distance module
• Multi-level alignment strategy for text guidance