Interpretable Multimodal Cancer Prototyping with Whole Slide Images and Incompletely Paired Genomics

2025-11-26

AI Summary

In clinical practice, incomplete or unavailable genomic data severely hinder multimodal precision oncology subtyping. To address this, we propose an interpretable fusion framework integrating whole-slide images (WSIs) with incompletely paired genomic data. Methodologically, our approach introduces biologically grounded prototypical representation, multi-view distribution- and sample-level alignment, bipartite fusion of shared and modality-specific features, and a semantic-prior-guided genomic imputation mechanism. This enables cross-modal collaborative learning and robust inference under missing modalities, significantly enhancing model interpretability and clinical applicability. Extensive experiments across multiple cancer typing and subtyping tasks—as well as genomic data imputation—demonstrate consistent superiority over state-of-the-art methods, validating the framework’s effectiveness, generalizability, and robustness to data incompleteness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Representation, semantic annotation, enhancement, enrichments, access and/or integration of a variety of data on the Web

Abstract

Multimodal approaches that integrate histology and genomics hold strong potential for precision oncology. However, phenotypic and genotypic heterogeneity limits the quality of intra-modal representations and hinders effective inter-modal integration. Furthermore, most existing methods overlook real-world clinical scenarios where genomics may be partially missing or entirely unavailable. We propose a flexible multimodal prototyping framework to integrate whole slide images and incomplete genomics for precision oncology. Our approach has four key components: 1) Biological Prototyping using text prompting and prototype-wise weighting; 2) Multiview Alignment through sample- and distribution-wise alignments; 3) Bipartite Fusion to capture both shared and modality-specific information for multimodal fusion; and 4) Semantic Genomics Imputation to handle missing data. Extensive experiments demonstrate the consistent superiority of the proposed method compared to other state-of-the-art approaches on multiple downstream tasks. The code is available at https://github.com/helenypzhang/Interpretable-Multimodal-Prototyping.

Problem

• Integrates histology and genomics for precision oncology despite data heterogeneity
• Handles incomplete or missing genomic data in real-world clinical scenarios
• Creates interpretable multimodal representations from whole slide images and genomics

Innovation

• Biological Prototyping with text prompting and prototype-wise weighting
• Multiview Alignment through sample- and distribution-wise alignments
• Semantic Genomics Imputation to handle missing data