How Good is my Histopathology Vision-Language Foundation Model? A Holistic Benchmark

2025-03-17

AI Summary

Existing histopathological vision-language models (VLMs) lack a unified, multi-center, multi-instrument, and multi-organ evaluation benchmark; publicly available data are limited by privacy constraints and modality scarcity. Method: We introduce HistoVL—the first open-source, comprehensive benchmark for histopathological VLMs—encompassing 26 organs, 31 cancer types, 14 patient cohorts, and over 5 million whole-slide image (WSI) patches, enabling fine-grained image-text pair construction and standardized downstream tasks (e.g., metastasis detection, organ classification). Contribution/Results: Using HistoVL, we systematically uncover critical clinical deployment bottlenecks in state-of-the-art pathological VLMs: high sensitivity to textual perturbations (25% accuracy drop), poor calibration (elevated expected calibration error), and adversarial vulnerability. HistoVL provides a reproducible, diagnostic evaluation framework to guide model optimization and trustworthy clinical deployment.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Recently, histopathology vision-language foundation models (VLMs) have gained popularity due to their enhanced performance and generalizability across different downstream tasks. However, most existing histopathology benchmarks are either unimodal or limited in terms of diversity of clinical tasks, organs, and acquisition instruments, as well as their partial availability to the public due to patient data privacy. As a consequence, there is a lack of comprehensive evaluation of existing histopathology VLMs on a unified benchmark setting that better reflects a wide range of clinical scenarios. To address this gap, we introduce HistoVL, a fully open-source comprehensive benchmark comprising images acquired using up to 11 various acquisition tools that are paired with specifically crafted captions by incorporating class names and diverse pathology descriptions. Our Histo-VL includes 26 organs, 31 cancer types, and a wide variety of tissue obtained from 14 heterogeneous patient cohorts, totaling more than 5 million patches obtained from over 41K WSIs viewed under various magnification levels. We systematically evaluate existing histopathology VLMs on Histo-VL to simulate diverse tasks performed by experts in real-world clinical scenarios. Our analysis reveals interesting findings, including large sensitivity of most existing histopathology VLMs to textual changes with a drop in balanced accuracy of up to 25% in tasks such as Metastasis detection, low robustness to adversarial attacks, as well as improper calibration of models evident through high ECE values and low model prediction confidence, all of which can affect their clinical implementation.

Problem

• Lack of comprehensive evaluation of histopathology vision-language models.
• Need for a unified benchmark reflecting diverse clinical scenarios.
• Existing models show sensitivity to textual changes and low robustness.

Innovation

• HistoVL benchmark with diverse clinical scenarios
• Includes 26 organs and 31 cancer types
• Evaluates VLMs on real-world clinical tasks