Multimodal Deep Learning for Subtype Classification in Breast Cancer Using Histopathological Images and Gene Expression Data

2025-03-04

AI Summary

Accurate molecular subtyping of breast cancer (Luminal vs. Basal/HER2) is critical for personalized therapy, yet single-modality approaches—relying solely on histopathological images or gene expression profiles—exhibit limited discriminative capacity. To address this, we propose an end-to-end deep multimodal classification framework that jointly models whole-slide histopathology images and bulk tumor gene expression data. Our method employs ResNet-50 for image feature extraction and a fully connected network for gene expression encoding, coupled with a novel cross-modal cross-attention fusion module that enables fine-grained semantic alignment and complementary representation learning. Evaluated on a multi-institutional cohort via five-fold cross-validation, the framework significantly outperforms unimodal baselines across accuracy, PR-AUC, and F1-score, while demonstrating robustness to inter-site variability and offering interpretable, biologically plausible attention patterns. These results validate the efficacy and clinical potential of synergistic multimodal integration in precision oncology.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Molecular subtyping of breast cancer is crucial for personalized treatment and prognosis. Traditional classification approaches rely on either histopathological images or gene expression profiling, limiting their predictive power. In this study, we propose a deep multimodal learning framework that integrates histopathological images and gene expression data to classify breast cancer into BRCA.Luminal and BRCA.Basal / Her2 subtypes. Our approach employs a ResNet-50 model for image feature extraction and fully connected layers for gene expression processing, with a cross-attention fusion mechanism to enhance modality interaction. We conduct extensive experiments using five-fold cross-validation, demonstrating that our multimodal integration outperforms unimodal approaches in terms of classification accuracy, precision-recall AUC, and F1-score. Our findings highlight the potential of deep learning for robust and interpretable breast cancer subtype classification, paving the way for improved clinical decision-making.

Problem

• Classify breast cancer subtypes using multimodal data.
• Integrate histopathological images and gene expression for classification.
• Improve classification accuracy with deep multimodal learning.

Innovation

• ResNet-50 extracts histopathological image features
• Fully connected layers process gene expression data
• Cross-attention fusion enhances modality interaction