Auditing Data Leakage in Whole-Slide Image Multimodal Benchmarks

2026-07-13

AI Summary

Current whole-slide image (WSI) multimodal visual question answering (VQA) benchmarks are severely compromised by patient- and institution-level data leakage, leading to inflated estimates of model reasoning capabilities. This work presents the first systematic audit of dual leakage issues in publicly available WSI VQA datasets. Through identifier tracing, linear separability analysis in feature space, and performance comparisons between leaked and clean samples, we reveal case overlap rates of 92.3%–100% in TCGA-derived benchmarks and demonstrate that leakage signals can be linearly decoded from features extracted by foundation models. Our findings indicate that reported high accuracies primarily stem from memorization of leakage artifacts rather than genuine multimodal reasoning. Based on these insights, we propose concrete guidelines for constructing contamination-free evaluation protocols.

Abstract

Recent vision-language models (VLMs) for computational pathology report striking zero-shot performance on whole-slide image (WSI) visual question answering (VQA) benchmarks. We audit these claims and find them fundamentally compromised by data leakage at two hierarchical levels: patient-level leakage, where slides from the same case appear in both training and test folds, and institutional-level leakage, where different cases nonetheless share staining-batch and scanner signatures through a common Tissue Source Site (TSS). By tracing canonical slide, case, and TSS identifiers across major public resources, we document case level train test overlaps of 92.3~100% on TCGA-derived benchmarks, together with near-complete TSS overlap. We further demonstrate that both leakage levels are linearly decodable from foundation-model feature space, that they induce a measurable accuracy gap between leaked and audit-clean cases on a published checkpoint, and that across multiple published WSI VLMs, peak reported accuracies concentrate on the most heavily contaminated benchmarks. Therefore, the current WSI VQA evaluation cannot distinguish genuine multimodal reasoning from nearest-neighbor retrieval over memorized institutional and patient-specific artifacts. Finally, we outline concrete recommendations for contamination-free evaluation. By addressing benchmark construction, provenance disclosure, and automated overlap auditing, we aim to guide future research toward verifiable claims of progress.

Problem

• data leakage
• whole-slide image
• visual question answering
• computational pathology
• benchmark contamination

Innovation

• data leakage
• whole-slide image
• vision-language models
• benchmark contamination
• computational pathology