A Geometric Multimodal Foundation Model Integrating Bp-MRI and Clinical Reports in Prostate Cancer Classification

2026-01-30

AI Summary

This work addresses the heavy reliance on subjective expert judgment in prostate cancer diagnosis and the limitations of existing computer-aided methods, which often neglect clinical context and suffer from scarce annotated data. To overcome these challenges, the authors propose MFM-Geom, a geometric multimodal foundation model that uniquely integrates Riemannian deep learning with multimodal foundation modeling. By jointly embedding biparametric MRI and clinical text on the manifold of symmetric positive definite (SPD) matrices, the method achieves geometric fusion of imaging and clinical information. Remarkably, MFM-Geom attains an AUC-PR of 90.67% using only 10% of the training data and maintains strong performance with 90.6% AUC-PR on an external test set, significantly outperforming current baselines and demonstrating exceptional few-shot generalization capability.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Prostate cancer (PCa) is one of the most common cancers in men worldwide. Bi-parametric MRI (bp-MRI) and clinical variables are crucial for PCa identification and improving treatment decisions. However, this process is subjective to expert interpretations. Furthermore, most existing computer-aided diagnosis methods focus on imaging-based models, overlooking the clinical context and suffering from data scarcity, limiting their ability to learn robust representations. We propose a geometric multimodal Foundation Model (FM), named MFM-Geom, that learns representations from bp-MRI and clinical reports, encoding visual findings and information from the context of clinical variables. In the representations classification head, the approach leverages symmetric positive definite (SPD) matrices and Riemannian deep learning to integrate imaging-text representations from a biomedical multimodal FM. Using 10% of the training data, MFM-Geom outperformed baseline class token embedding-based classification (+8.3%, AUC-PR of 90.67). Generalization on external dataset confirmed the robustness of fine-tuning biomedical FM, achieving an AUC-PR of 90.6.

Problem

• prostate cancer
• bp-MRI
• clinical reports
• multimodal learning
• data scarcity

Innovation

• geometric multimodal foundation model
• Riemannian deep learning
• symmetric positive definite matrices
• bp-MRI
• clinical reports