Semantic-guided Masked Mutual Learning for Multi-modal Brain Tumor Segmentation with Arbitrary Missing Modalities

2025-07-10

AI Summary

Multi-modal brain tumor segmentation suffers from significant performance degradation under arbitrary missing modalities. To address this, we propose Semantic-Guided Mask Mutual Learning (SG-MML), a dual-branch framework that enables cross-modal mask mutual distillation via pixel-level reliable knowledge exchange and feature-level relational modeling. Hierarchical consistency constraints—enforced jointly in mask, feature, and relational spaces—ensure multi-granularity semantic alignment. Furthermore, SG-MML incorporates strong semantic priors from the Segment Anything Model (SAM) to enhance discriminative capability under modality absence. Evaluated on BraTS2018, BraTS2020, and BraTS2021 benchmarks, SG-MML achieves state-of-the-art performance across single-modality, dual-modality, and randomly missing modality settings. It notably improves segmentation robustness and generalizability under incomplete input conditions, demonstrating superior resilience to modality dropout compared to existing methods.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

Malignant brain tumors have become an aggressive and dangerous disease that leads to death worldwide.Multi-modal MRI data is crucial for accurate brain tumor segmentation, but missing modalities common in clinical practice can severely degrade the segmentation performance. While incomplete multi-modal learning methods attempt to address this, learning robust and discriminative features from arbitrary missing modalities remains challenging. To address this challenge, we propose a novel Semantic-guided Masked Mutual Learning (SMML) approach to distill robust and discriminative knowledge across diverse missing modality scenarios.Specifically, we propose a novel dual-branch masked mutual learning scheme guided by Hierarchical Consistency Constraints (HCC) to ensure multi-level consistency, thereby enhancing mutual learning in incomplete multi-modal scenarios. The HCC framework comprises a pixel-level constraint that selects and exchanges reliable knowledge to guide the mutual learning process. Additionally, it includes a feature-level constraint that uncovers robust inter-sample and inter-class relational knowledge within the latent feature space. To further enhance multi-modal learning from missing modality data, we integrate a refinement network into each student branch. This network leverages semantic priors from the Segment Anything Model (SAM) to provide supplementary information, effectively complementing the masked mutual learning strategy in capturing auxiliary discriminative knowledge. Extensive experiments on three challenging brain tumor segmentation datasets demonstrate that our method significantly improves performance over state-of-the-art methods in diverse missing modality settings.

Problem

• Segmentation of brain tumors with missing MRI modalities
• Learning robust features from arbitrary missing modalities
• Enhancing multi-modal consistency and discriminative knowledge

Innovation

• Semantic-guided masked mutual learning scheme
• Hierarchical Consistency Constraints for multi-level consistency
• Segment Anything Model for supplementary semantic priors