PROTON: Prototype-Based Test-Time Online OOD Detection for Medical VLMs

2026-06-18

AI Summary

This work addresses the challenge of reliably detecting out-of-distribution (OOD) inputs in deployed medical vision-language models (VLMs), particularly under covariate shift where existing static scoring methods fail. To this end, the authors propose PROTON, a lightweight post-processing module that requires no modifications to the model architecture, training data, or prompt engineering. PROTON dynamically constructs an online repository of high-confidence sample prototypes and adaptively fuses prototype distances with maximum concept matching scores. It is the first method to exploit the structural properties of covariate-shifted samples in the VLM embedding space, establishing a unified, zero-shot OOD detection framework without relying on labels or hierarchical prompting. On the FLAIR+FIVES ophthalmic benchmark, PROTON improves AUROC by 23.9%, 8.8%, and 8.1% under covariate shift, semantic shift, and far-OOD settings, respectively, and stands as the only zero-shot approach achieving consistent gains across all shift types.

Abstract

Medical vision-language models (VLMs) enable zero-shot clinical image classification, yet reliably detecting out-of-distribution (OOD) inputs at deployment remains an open problem. No static scoring method works across all shift types: Maximum Concept Matching (MCM) on FLAIR achieves 76.4% AUROC for far-OOD but only 42.4% for covariate shifts such as ultra-wide-field fundus images, effectively random. We trace this to a structural mismatch: covariate-shifted inputs are indistinguishable from in-distribution samples in softmax space, yet occupy distinct regions in the VLM embedding space. To exploit this untapped signal, we propose PROTON (PROtotype-based Test-time ONline OOD detection), a lightweight post-hoc module that maintains an online prototype bank from high-confidence test predictions and adaptively fuses prototype distance with MCM scoring via stream-level variance statistics, requiring no model modification, training data, or prompt engineering. On the ophthalmology benchmark FLAIR + FIVES, PROTON improves MCM by +23.9 AUROC on covariate shift, +8.8 on semantic shift, and +8.1 on far-OOD, making it the only zero-shot method to improve all three without hierarchical prompts or labeled data. Code is available at https://github.com/GenMI-Lab/PROTON, and the project page is available at https://genmi-lab.github.io/PROTON.

Problem

• out-of-distribution detection
• medical vision-language models
• covariate shift
• zero-shot classification
• deployment robustness

Innovation

• prototype-based
• test-time adaptation
• out-of-distribution detection
• medical vision-language models
• zero-shot learning