A Brain Tumor Segmentation Method Based on CLIP and 3D U-Net with Cross-Modal Semantic Guidance and Multi-Level Feature Fusion

2025-07-14

AI Summary

Brain tumor MRI segmentation faces challenges from high morphological heterogeneity, complex 3D spatial relationships, and the neglect of semantic knowledge embedded in clinical radiology reports. Method: This paper proposes a multi-level cross-modal fusion framework that—novelty—introduces CLIP into 3D brain tumor segmentation. It constructs a 3D–2D semantic bridging module and designs cross-modal semantic guidance and semantic attention mechanisms to jointly model pixel-level, feature-level, and semantic-level information. The framework seamlessly integrates CLIP’s semantic understanding capability with 3D U-Net’s spatial representation strength. Results: On the BraTS 2020 dataset, the method achieves an overall Dice score of 0.8567—4.8% higher than the 3D U-Net baseline—and improves the Dice score for the enhancing tumor subregion by 7.3%, significantly enhancing robustness in identifying complex lesions.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Precise segmentation of brain tumors from magnetic resonance imaging (MRI) is essential for neuro-oncology diagnosis and treatment planning. Despite advances in deep learning methods, automatic segmentation remains challenging due to tumor morphological heterogeneity and complex three-dimensional spatial relationships. Current techniques primarily rely on visual features extracted from MRI sequences while underutilizing semantic knowledge embedded in medical reports. This research presents a multi-level fusion architecture that integrates pixel-level, feature-level, and semantic-level information, facilitating comprehensive processing from low-level data to high-level concepts. The semantic-level fusion pathway combines the semantic understanding capabilities of Contrastive Language-Image Pre-training (CLIP) models with the spatial feature extraction advantages of 3D U-Net through three mechanisms: 3D-2D semantic bridging, cross-modal semantic guidance, and semantic-based attention mechanisms. Experimental validation on the BraTS 2020 dataset demonstrates that the proposed model achieves an overall Dice coefficient of 0.8567, representing a 4.8% improvement compared to traditional 3D U-Net, with a 7.3% Dice coefficient increase in the clinically important enhancing tumor (ET) region.

Problem

• Precise brain tumor segmentation from MRI scans
• Integrating semantic knowledge with visual MRI features
• Improving 3D U-Net performance via cross-modal fusion

Innovation

• Combines CLIP and 3D U-Net for segmentation
• Uses cross-modal semantic guidance mechanisms
• Implements multi-level feature fusion architecture