Pretraining Multiple Instance Learning Networks with Multi-Teacher Distillation from Pathology Slide Foundation Models

2026-07-16

AI Summary

This work addresses the limitations of existing multiple instance learning (MIL) models in whole-slide image analysis, which are typically trained from scratch and suffer from optimization instability, overfitting, and poor generalization due to sparse slide-level labels. To overcome these challenges, the authors propose the first general-purpose pretraining framework for MIL, leveraging multi-teacher knowledge distillation with two foundational pathology models—TITAN and CARE—as teachers to transfer rich representational knowledge to diverse MIL architectures. They introduce an angular dispersion-normalized distillation loss to effectively balance supervision signals from multiple sources. Extensive experiments across 15 benchmark datasets demonstrate that this pretraining strategy substantially outperforms training from scratch, yielding notable gains especially in linear probing and few-shot settings, while preserving the computational efficiency of lightweight MIL models.

Abstract

Multiple instance learning (MIL) has become the main paradigm for whole-slide image (WSI) analysis in computational pathology. However, existing MIL aggregators are still typically trained from scratch for each downstream task, relying on limited slide-level labels to learn both aggregation mechanisms and downstream discriminative representations simultaneously. As a result, they often suffer from unstable optimization, overfitting, and limited transferability. Similar to pretrained ResNet and Vision Transformer models in natural image learning, MIL also requires reusable pretrained initialization. However, high-quality slide-level pretraining data remain scarce, and MIL models are usually lightweight and weakly supervised, making large-scale pretraining difficult in practice. To address this challenge, we propose a distillation-based pretraining framework for MIL, which leverages two slide-level foundation models, TITAN and CARE, as teachers to transfer their representational knowledge into a diverse set of MIL architectures. To effectively balance supervision from different teachers, we further introduce an angular dispersion normalized distillation loss. The distilled weights are then used as initialization for downstream adaptation. We conduct systematic evaluations on 15 benchmark datasets under both linear probing and full-parameter fine-tuning, and further validate its advantages in few-shot scenarios. Experimental results show that pretraining generally improves MIL aggregators over from scratch training, especially in linear-probing and few-shot settings, while maintaining the computational efficiency of lightweight MIL models. Code is available at https://github.com/fu0201/MIL_Pretrained.

Problem

• Multiple Instance Learning
• Whole-slide Image
• Pretraining
• Computational Pathology
• Transferability

Innovation

• multiple instance learning
• knowledge distillation
• foundation models
• pretraining
• computational pathology