ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification

2024-06-16

Computer Vision and Pattern Recognition

AI Summary

In whole-slide image (WSI) classification for digital pathology, existing multiple instance learning (MIL) methods suffer from heavy reliance on abundant bag-level annotations and poor generalizability, while vision-language models (VLMs) are hindered by pathology-agnostic text prompts and prohibitively high pretraining costs, yielding limited performance gains. To address these limitations, we propose ViLa-MIL—a dual-scale vision-language MIL framework. It introduces the first pathology-informed, dual-scale descriptive text prompting mechanism; designs a prototype-guided patch decoder and a context-guided text decoder to enable cross-modal, multi-granularity feature co-modeling; and integrates a frozen large language model, prototype clustering, and vision-language alignment. Evaluated on three multi-cancer, multi-center datasets, ViLa-MIL significantly outperforms state-of-the-art methods, demonstrating low annotation dependency, strong cross-center generalizability, and high robustness.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI) with giga-pixel size and hierarchical image context in digital pathology. However, these methods heavily depend on a substantial number of bag-level labels and solely learn from the original slides, which are easily affected by variations in data distribution. Recently, vision language model (VLM)-based methods introduced the language prior by pre-training on large-scale pathological image-text pairs. However, the previous text prompt lacks the consideration of pathological prior knowledge, there-fore does not substantially boost the model's performance. Moreover, the collection of such pairs and the pre-training process are very time-consuming and source-intensive. To solve the above problems, we propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification. Specifically, we propose a dual-scale visual descriptive text prompt based on the frozen large language model (LLM) to boost the performance of VLM effectively. To transfer the VLM to process WSI efficiently, for the image branch, we propose a prototype-guided patch decoder to aggregate the patch features progressively by grouping similar patches into the same prototype; for the text branch, we introduce a context-guided text decoder to enhance the text features by incorporating the multi-granular image contexts. Extensive studies on three multi-cancer and multi-center subtyping datasets demonstrate the superiority of ViLa-MIL.

Problem

• Enhance WSI classification using dual-scale MIL.
• Integrate pathological knowledge into text prompts.
• Optimize resource use in VLM pre-training.

Innovation

• Dual-scale vision-language multiple instance learning
• Prototype-guided patch feature aggregation
• Context-guided text feature enhancement