Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology

2026-05-18

AI Summary

This work addresses the high sensitivity of vision-language models to example selection and prompt phrasing in in-context learning for computational pathology, which undermines diagnostic reliability. To mitigate this issue, the authors propose a training-free, geometry-aware uncertainty-based coreset selection method that jointly optimizes distributional fidelity, prompt robustness, and prediction stability within the pretrained multimodal embedding space. Their approach uniquely integrates maximum mean discrepancy, an effective mutual information difference regularizer, and a prediction variance penalty—significantly enhancing the reliability and calibration of in-context learning without updating model parameters. Experiments on the CRC-100K and MHIST datasets demonstrate consistent improvements over existing in-context example selection strategies and data distillation baselines in terms of accuracy, calibration, and robustness to prompt variations.

Abstract

Vision-language models (VLMs) can couple visual perception with open-ended clinical reasoning, making them attractive for computational histopathology. However, fine-tuning billions of parameters on scarce, expert-annotated pathology data is prohibitive, while in-context learning (ICL), which conditions the VLM on demonstrative image-text pairs without parameter updates, suffers from high sensitivity to which examples are selected and how the query is phrased, producing unreliable diagnostics. Existing selection strategies rely on query-dependent nearest-neighbour retrieval that ignores global data structure, require costly parameter updates, or disregard the joint vision-text embedding geometry of VLMs. We propose GAUC, a training-free coreset selection method operating directly in the pre-trained multimodal embedding space. GAUC jointly optimises three objectives: (1) a Maximum Mean Discrepancy term enforcing distributional fidelity between coreset and full dataset, (2) an Effective Mutual Information Difference regulariser bounding performance degradation under prompt paraphrases by exploiting the VLM's joint vision-text alignment, and (3) a predictive-variance penalty suppressing overconfident, unstable outputs. On CRC-100K and MHIST across multiple open-source VLM architectures, GAUC consistently improves accuracy, calibration, and prompt robustness over recent ICL selection methods and dataset-distillation baselines, all without a single gradient update.

Problem

• in-context learning
• visual-language models
• histopathology
• coreset selection
• embedding geometry

Innovation

• Geometry-Aware Uncertainty Coresets
• In-Context Learning
• Vision-Language Models
• Maximum Mean Discrepancy
• Prompt Robustness