TGC-Net: A Structure-Aware and Semantically-Aligned Framework for Text-Guided Medical Image Segmentation

2025-12-24

AI Summary

To address three key challenges in clinical-report-guided medical image segmentation—cross-modal semantic misalignment, loss of anatomical details, and domain-specific bias—this paper proposes a lightweight, CLIP-based framework. Methodologically, we design a semantic-structural co-encoder to jointly model clinical descriptions and preserve fine-grained anatomical structures; introduce a domain-enhanced text encoder that integrates large-model-driven medical terminology understanding; and construct a vision-language calibration module to align cross-modal feature spaces. Technically, we adopt a ViT-CNN dual-path image encoder and parameter-efficient fine-tuning. Evaluated on five X-ray and CT datasets, our method achieves state-of-the-art performance, improving mean Dice score by 3.2% while reducing trainable parameters by 41.7%.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Text-guided medical segmentation enhances segmentation accuracy by utilizing clinical reports as auxiliary information. However, existing methods typically rely on unaligned image and text encoders, which necessitate complex interaction modules for multimodal fusion. While CLIP provides a pre-aligned multimodal feature space, its direct application to medical imaging is limited by three main issues: insufficient preservation of fine-grained anatomical structures, inadequate modeling of complex clinical descriptions, and domain-specific semantic misalignment. To tackle these challenges, we propose TGC-Net, a CLIP-based framework focusing on parameter-efficient, task-specific adaptations. Specifically, it incorporates a Semantic-Structural Synergy Encoder (SSE) that augments CLIP's ViT with a CNN branch for multi-scale structural refinement, a Domain-Augmented Text Encoder (DATE) that injects large-language-model-derived medical knowledge, and a Vision-Language Calibration Module (VLCM) that refines cross-modal correspondence in a unified feature space. Experiments on five datasets across chest X-ray and thoracic CT modalities demonstrate that TGC-Net achieves state-of-the-art performance with substantially fewer trainable parameters, including notable Dice gains on challenging benchmarks.

Problem

• Enhances segmentation using clinical reports as auxiliary information
• Addresses CLIP's limitations in medical imaging for fine-grained structures
• Reduces trainable parameters while improving cross-modal alignment accuracy

Innovation

• CLIP-based framework with parameter-efficient task adaptations
• Semantic-Structural Synergy Encoder for multi-scale anatomical refinement
• Vision-Language Calibration Module for unified cross-modal alignment