Towards a Multimodal MRI-Based Foundation Model for Multi-Level Feature Exploration in Segmentation, Molecular Subtyping, and Grading of Glioma

2025-03-10

AI Summary

To address insufficient modeling of spatial heterogeneity and challenges in multi-task coordination for glioma diagnosis, this paper proposes a multitask SWIN-UNETR (MTS-UNET) foundation model. Built upon the BrainSegFounder framework, it is pretrained on 2,249 multicenter MRI cases from seven public cohorts. For the first time, it integrates tumor-aware feature encoding (TAFE) with a cross-modal discrepancy module (CMD) to jointly model morphological heterogeneity and IDH-associated T2-FLAIR mismatch signals. The model simultaneously achieves glioma segmentation (Dice score: 84%), IDH mutation prediction (AUC: 90.58%), 1p/19q codeletion classification (AUC: 69.22%), and histopathological grading (AUC: 87.54%), significantly outperforming baseline methods (p ≤ 0.05). It demonstrates robust generalizability across independent centers and overcomes limitations of conventional invasive diagnostic approaches.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Accurate, noninvasive glioma characterization is crucial for effective clinical management. Traditional methods, dependent on invasive tissue sampling, often fail to capture the spatial heterogeneity of the tumor. While deep learning has improved segmentation and molecular profiling, few approaches simultaneously integrate tumor morphology and molecular features. Foundation deep learning models, which learn robust, task-agnostic representations from large-scale datasets, hold great promise but remain underutilized in glioma imaging biomarkers. We propose the Multi-Task SWIN-UNETR (MTS-UNET) model, a novel foundation-based framework built on the BrainSegFounder model, pretrained on large-scale neuroimaging data. MTS-UNET simultaneously performs glioma segmentation, histological grading, and molecular subtyping (IDH mutation and 1p/19q co-deletion). It incorporates two key modules: Tumor-Aware Feature Encoding (TAFE) for multi-scale, tumor-focused feature extraction and Cross-Modality Differential (CMD) for highlighting subtle T2-FLAIR mismatch signals associated with IDH mutation. The model was trained and validated on a diverse, multi-center cohort of 2,249 glioma patients from seven public datasets. MTS-UNET achieved a mean Dice score of 84% for segmentation, along with AUCs of 90.58% for IDH mutation, 69.22% for 1p/19q co-deletion prediction, and 87.54% for grading, significantly outperforming baseline models (p<=0.05). Ablation studies validated the essential contributions of the TAFE and CMD modules and demonstrated the robustness of the framework. The foundation-based MTS-UNET model effectively integrates tumor segmentation with multi-level classification, exhibiting strong generalizability across diverse MRI datasets. This framework shows significant potential for advancing noninvasive, personalized glioma management by improving predictive accuracy and interpretability.

Problem

• Accurate noninvasive glioma characterization for clinical management
• Integrating tumor morphology and molecular features simultaneously
• Developing a foundation model for multi-level glioma analysis

Innovation

• Multi-Task SWIN-UNETR model for glioma analysis
• Tumor-Aware Feature Encoding for multi-scale extraction
• Cross-Modality Differential for IDH mutation signals