AdaFusion: Prompt-Guided Inference with Adaptive Fusion of Pathology Foundation Models

2025-08-07

AI Summary

Pathological foundation models (PFMs) suffer from limited generalizability and interpretability due to biases in pretraining data and architectural design. To address this, we propose the first prompt-guided dynamic multi-PFM fusion framework, which enables tissue-phenotype-aware adaptive inference via feature compression-based alignment and a lightweight attention mechanism—explicitly revealing each model’s biologically grounded semantic expertise. Our method jointly optimizes performance and decision transparency. Evaluated on three real-world downstream tasks—treatment response prediction, tumor grading, and spatial gene expression inference—it consistently outperforms individual PFMs, achieving state-of-the-art results across both classification and regression metrics. This work establishes a scalable, interpretable paradigm for collaborative multi-model deployment of PFMs, advancing trustworthy clinical AI.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Pathology foundation models (PFMs) have demonstrated strong representational capabilities through self-supervised pre-training on large-scale, unannotated histopathology image datasets. However, their diverse yet opaque pretraining contexts, shaped by both data-related and structural/training factors, introduce latent biases that hinder generalisability and transparency in downstream applications. In this paper, we propose AdaFusion, a novel prompt-guided inference framework that, to our knowledge, is among the very first to dynamically integrate complementary knowledge from multiple PFMs. Our method compresses and aligns tile-level features from diverse models and employs a lightweight attention mechanism to adaptively fuse them based on tissue phenotype context. We evaluate AdaFusion on three real-world benchmarks spanning treatment response prediction, tumour grading, and spatial gene expression inference. Our approach consistently surpasses individual PFMs across both classification and regression tasks, while offering interpretable insights into each model's biosemantic specialisation. These results highlight AdaFusion's ability to bridge heterogeneous PFMs, achieving both enhanced performance and interpretability of model-specific inductive biases.

Problem

• Address latent biases in pathology foundation models
• Dynamically integrate knowledge from multiple PFMs
• Enhance performance and interpretability in downstream tasks

Innovation

• Prompt-guided dynamic fusion of multiple PFMs
• Lightweight attention for adaptive feature fusion
• Tile-level feature compression and alignment