Shazam: Unifying Multiple Foundation Models for Advanced Computational Pathology

2025-03-02

AI Summary

In computational pathology, existing foundation models (FMs) suffer from poor generalizability and unstable performance when deployed across diverse downstream tasks. To address this, we propose Shazam—a lightweight, multi-model collaborative inference framework for multi-task adaptation. Its core innovation is the first-of-its-kind multi-teacher collaborative distillation mechanism, which employs dynamic weight scheduling and multi-source feature fusion to prevent dominance by any single model, thereby enabling task-driven feature refinement and complementary knowledge integration. Crucially, Shazam operates without fine-tuning individual FMs, significantly reducing computational overhead. Evaluated on two benchmark histopathological image patch classification datasets, Shazam consistently outperforms both individual models and state-of-the-art ensemble methods, achieving superior accuracy while reducing inference cost by over 30%. This establishes a robust, scalable, and unified inference paradigm for clinical pathology AI.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Foundation Models (FMs) in computational pathology (CPath) have significantly advanced the extraction of meaningful features from histopathology image datasets, achieving strong performance across various clinical tasks. Despite their impressive performance, these models often exhibit variability when applied to different tasks, prompting the need for a unified framework capable of consistently excelling across various applications. In this work, we propose Shazam, a novel framework designed to efficiently combine multiple CPath models. Unlike previous approaches that train a fixed-parameter FM, Shazam dynamically extracts and refines information from diverse FMs for each specific task. To ensure that each FM contributes effectively without dominance, a novel distillation strategy is applied, guiding the student model with features from all teacher models, which enhances its generalization ability. Experimental results on two pathology patch classification datasets demonstrate that Shazam outperforms existing CPath models and other fusion methods. Its lightweight, flexible design makes it a promising solution for improving CPath analysis in real-world settings. Code will be available at https://github.com/Tuner12/Shazam.

Problem

• Unifies multiple foundation models for computational pathology tasks.
• Addresses variability in model performance across different clinical applications.
• Enhances generalization through a novel distillation strategy.

Innovation

• Dynamic combination of multiple foundation models
• Novel distillation strategy for model integration
• Lightweight and flexible design for real-world applications