Probing, Fusion, and Trustworthiness: A Systematic Evaluation of Foundation Model Representations for Multimodal Cancer Analysis

2026-06-15

AI Summary

This study evaluates the generalization capability and clinical reliability of foundation models under distribution shifts using multimodal cancer data, specifically whole-slide images and transcriptomic profiles. Through unimodal probing, multimodal fusion, and conformal prediction, the work systematically examines the complementarity between imaging and omics representations and validates the efficacy of fusion strategies in scenarios where no single modality dominates. The results demonstrate that foundation models exhibit robust performance on out-of-distribution data, that multimodal fusion significantly enhances predictive accuracy, and that incorporating conformal prediction substantially improves diagnostic recoverability and clinical trustworthiness.

Abstract

Foundation models (FMs) have emerged as powerful representation extractors for medical data, yet their generalizability to datasets under distribution shift remains underexplored. This work systematically evaluates FM-based representations on a suite of computational pathology tasks across two real-world commercial cohorts, IH-BC and IH-NSCLC, drawn from the licensed in-house (IH) oncology dataset. The analysis focuses on two modalities, whole-slide images and transcriptomic profiles, drawn from the IH multimodal data. We first benchmark unimodal probing performance across five FMs on eight downstream classification tasks, and find that image and omics representations carry complementary predictive signals. Then we investigate whether multimodal fusion can yield additional gains over unimodal baselines by comparing three image-omics fusion strategies built on paired representations. The trustworthiness of selected unimodal and multimodal pipelines is further assessed through conformal prediction. Our results show that FM representations achieve competitive performance on out-of-distribution data and that multimodal fusion helps mainly when no single modality dominates the signal. Conformal prediction reveals that in the majority of cases where a point prediction fails, the true diagnosis remains recoverable within the prediction set, reinforcing the value of uncertainty-aware inference for clinical support.

Problem

• foundation models
• multimodal fusion
• distribution shift
• trustworthiness
• cancer analysis

Innovation

• foundation models
• multimodal fusion
• conformal prediction
• distribution shift
• computational pathology