Sub-Region-Aware Modality Fusion and Adaptive Prompting for Multi-Modal Brain Tumor Segmentation

2026-01-22

AI Summary

This work proposes a foundational model adaptation framework for brain tumor segmentation to address the limitations in segmentation accuracy caused by insufficient multimodal fusion and tumor heterogeneity. The approach introduces a subregion-aware modality attention mechanism that dynamically learns the optimal modality combination for each tumor subregion, coupled with adaptive prompt engineering to effectively leverage the prior knowledge embedded in the foundational model. Experimental results on the BraTS 2020 dataset demonstrate that the proposed method significantly outperforms existing baselines, particularly excelling in challenging subregions such as the necrotic core, thereby validating its effectiveness and robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The successful adaptation of foundation models to multi-modal medical imaging is a critical yet unresolved challenge. Existing models often struggle to effectively fuse information from multiple sources and adapt to the heterogeneous nature of pathological tissues. To address this, we introduce a novel framework for adapting foundation models to multi-modal medical imaging, featuring two key technical innovations: sub-region-aware modality attention and adaptive prompt engineering. The attention mechanism enables the model to learn the optimal combination of modalities for each tumor sub-region, while the adaptive prompting strategy leverages the inherent capabilities of foundation models to refine segmentation accuracy. We validate our framework on the BraTS 2020 brain tumor segmentation dataset, demonstrating that our approach significantly outperforms baseline methods, particularly in the challenging necrotic core sub-region. Our work provides a principled and effective approach to multi-modal fusion and prompting, paving the way for more accurate and robust foundation model-based solutions in medical imaging.

Problem

• multi-modal fusion
• brain tumor segmentation
• foundation models
• pathological heterogeneity
• sub-region awareness

Innovation

• sub-region-aware modality fusion
• adaptive prompting
• foundation model adaptation
• multi-modal brain tumor segmentation
• modality attention