CmFNet: Cross-modal Fusion Network for Weakly-supervised Segmentation of Medical Images

2025-06-22

AI Summary

Weakly supervised medical image segmentation suffers from performance degradation and overfitting due to sparse annotations. To address this, we propose a 3D cross-modal weakly supervised segmentation framework for CT/MR multimodal imaging. Our method introduces two key innovations: (1) a modality-specific and cross-modal collaborative feature learning network, and (2) a hybrid supervision strategy integrating scribble annotations, intra-modal regularization, and inter-modal consistency constraints to enhance feature alignment and contextual modeling. Evaluated on nasopharyngeal carcinoma CT/MR and WORD abdominal CT datasets, our approach significantly outperforms existing weakly supervised methods. Remarkably, even under full-supervision settings, it surpasses standard fully supervised baselines—particularly improving segmentation accuracy for small tumors and fine anatomical structures.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Accurate automatic medical image segmentation relies on high-quality, dense annotations, which are costly and time-consuming. Weakly supervised learning provides a more efficient alternative by leveraging sparse and coarse annotations instead of dense, precise ones. However, segmentation performance degradation and overfitting caused by sparse annotations remain key challenges. To address these issues, we propose CmFNet, a novel 3D weakly supervised cross-modal medical image segmentation approach. CmFNet consists of three main components: a modality-specific feature learning network, a cross-modal feature learning network, and a hybrid-supervised learning strategy. Specifically, the modality-specific feature learning network and the cross-modal feature learning network effectively integrate complementary information from multi-modal images, enhancing shared features across modalities to improve segmentation performance. Additionally, the hybrid-supervised learning strategy guides segmentation through scribble supervision, intra-modal regularization, and inter-modal consistency, modeling spatial and contextual relationships while promoting feature alignment. Our approach effectively mitigates overfitting, delivering robust segmentation results. It excels in segmenting both challenging small tumor regions and common anatomical structures. Extensive experiments on a clinical cross-modal nasopharyngeal carcinoma (NPC) dataset (including CT and MR imaging) and the publicly available CT Whole Abdominal Organ dataset (WORD) show that our approach outperforms state-of-the-art weakly supervised methods. In addition, our approach also outperforms fully supervised methods when full annotation is used. Our approach can facilitate clinical therapy and benefit various specialists, including physicists, radiologists, pathologists, and oncologists.

Problem

• Reduces reliance on costly dense medical image annotations
• Improves weakly-supervised segmentation performance and prevents overfitting
• Enhances cross-modal feature fusion for accurate tumor and organ segmentation

Innovation

• Cross-modal fusion for multi-modal feature integration
• Hybrid-supervised learning with scribble and consistency
• 3D weakly supervised segmentation for medical images