Vision-Language Models as Zero-Annotation Oracles in Histopathology

2026-06-15

AI Summary

This study addresses the limitations of existing foreground segmentation methods in computational pathology—namely, their reliance on manual parameter tuning or supervised models and poor generalization to specialty-stained histopathology slides—by introducing, for the first time, a general-purpose vision-language model (VLM) for zero-shot, annotation-free foreground segmentation. The proposed approach employs a coarse-to-fine inference framework augmented with an Auto-context few-shot prompting mechanism to refine discrimination in challenging regions, followed by pseudo-label distillation to train a lightweight student model. Evaluated on the Leica-75 benchmark, the method achieves Dice scores of 0.858 and 0.853 for Jones and EVG stains, respectively, reducing cross-stain variance by sevenfold. VLM-generated annotations exhibit near-perfect agreement with expert annotations (Cohen’s κ = 0.989), and the distilled student model maintains high performance while substantially lowering computational cost.

Abstract

Foreground segmentation is the critical first step of every computational pathology pipeline, yet existing methods rely on hand-tuned heuristics or supervised models that overfit to narrow stain and scanner distributions, failing silently on specialised stains such as Jones silver or Elastica van Gieson. We propose a coarse-to-fine approach that recasts foreground segmentation as a visual perception task and leverages general-purpose vision-language models (VLMs) as zero-annotation oracles. Our key insight is that tissue-versus-background discrimination is a natural-image recognition problem, not a histopathological one, so VLMs trained on internet-scale corpora generalise where domain-specific models cannot. We introduce Leica-75, a benchmark of 75 renal transplant whole-slide images spanning three stain families. On Leica-75, our method achieves the highest segmentation quality on out-of-distribution stains (Dice 0.858 +/- 0.027 on Jones, 0.853 +/- 0.041 on EVG) with 7x lower cross-stain variance than the best supervised baseline, while remaining competitive on in-distribution H&E. Few-shot prompting with automatically curated exemplars (Auto-context) rescues hard cases on Stress-32 (n=32), a curated stress-test subset (Dice 0.470 to 0.819 for the 2B model). VLM-based annotation review matches human expert consensus (kappa=0.989 for blur detection; mean precision/recall grading accuracy 0.708 vs. human 0.646 for segmentation mask review). The resulting pseudo-labels are used to distil lightweight student models that are as performant as the teacher model while running for a fraction of the cost. Our framework provides a principled, scalable solution to a persistent infrastructure bottleneck in digital pathology.

Problem

• foreground segmentation
• computational pathology
• histopathology
• stain generalization
• domain shift

Innovation

• vision-language models
• zero-annotation
• foreground segmentation
• cross-stain generalization
• pseudo-label distillation