Towards A Generalizable Pathology Foundation Model via Unified Knowledge Distillation

2024-07-26

arXiv.org

AI Summary

Current foundational models in computational pathology lack systematic evaluation of generalization capability across diverse clinical tasks and tissue types. Method: We introduce the first comprehensive pathology generalization benchmark—covering 39 clinical tasks across six categories and 34 tissue types—to systematically identify generalization bottlenecks in state-of-the-art models. Leveraging this benchmark, we propose a unified knowledge distillation framework that jointly employs multi-expert guidance and self-distillation to align local and global representations, enabling the pretraining of a general-purpose foundational model, GPFM. GPFM is trained via large-scale self-supervised representation learning on over one million H&E-stained whole-slide images. Contribution/Results: GPFM achieves top-1 performance on 29 of the 39 tasks, with a mean rank of 1.36—significantly outperforming UNI (2.96). It establishes a new feature backbone for computational pathology, setting a benchmark for cross-task and cross-tissue generalization.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Computer Vision: Other Foundations of Computer Vision
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Foundation models pretrained on large-scale datasets are revolutionizing the field of computational pathology (CPath). The generalization ability of foundation models is crucial for the success in various downstream clinical tasks. However, current foundation models have only been evaluated on a limited type and number of tasks, leaving their generalization ability and overall performance unclear. To address this gap, we established a most comprehensive benchmark to evaluate the performance of off-the-shelf foundation models across six distinct clinical task types, encompassing a total of 39 specific tasks. Our findings reveal that existing foundation models excel at certain task types but struggle to effectively handle the full breadth of clinical tasks. To improve the generalization of pathology foundation models, we propose a unified knowledge distillation framework consisting of both expert and self knowledge distillation, where the former allows the model to learn from the knowledge of multiple expert models, while the latter leverages self-distillation to enable image representation learning via local-global alignment. Based on this framework, a Generalizable Pathology Foundation Model (GPFM) is pretrained on a large-scale dataset consisting of 190 million images from around 86,000 public H&E whole slides across 34 major tissue types. Evaluated on the established benchmark, GPFM achieves an impressive average rank of 1.36, with 29 tasks ranked 1st, while the the second-best model, UNI, attains an average rank of 2.96, with only 4 tasks ranked 1st. The superior generalization of GPFM demonstrates its exceptional modeling capabilities across a wide range of clinical tasks, positioning it as a new cornerstone for feature representation in CPath.

Problem

• Evaluating generalization of pathology foundation models across diverse clinical tasks
• Proposing unified knowledge distillation to enhance model adaptability
• Developing a comprehensive benchmark with 72 specific pathology tasks

Innovation

• Unified knowledge distillation framework
• Expert and self-knowledge distillation
• Generalizable Pathology Foundation Model