Respecting Modality Gap in Post-hoc Out-of-distribution Detection with Pre-trained Vision-Language Models

2026-05-26

AI Summary

Existing zero-shot out-of-distribution (OOD) detection methods directly employ text embeddings as class prototypes, overlooking the semantic gap between visual and linguistic modalities, which limits their performance. This work theoretically characterizes this modality gap for the first time and introduces an online pseudo-supervised framework that operates without in-distribution training data. During inference, the method leverages unlabeled data streams and soft predictions from a pretrained vision-language model to dynamically learn optimal class prototypes in the visual feature space. By challenging the conventional assumption that “text embeddings are prototypes,” the proposed approach achieves state-of-the-art performance across multiple OOD benchmarks, demonstrating both theoretical rigor and strong generalization capability.

Abstract

Out-of-distribution (OOD) detection has emerged as a popular technique to enhance the reliability of machine learning models by identifying unexpected inputs from unknown classes. Recent progress in pre-trained vision-language models (VLMs) has enabled zero-shot OOD detection without access to in-distribution (ID) training data; in this setting, existing methods commonly treat text embeddings of class names as class prototypes. In this paper, we challenge the widely adopted text-as-prototype paradigm by theoretically showing that off-the-shelf textual prototypes are generally misaligned with the optimal visual prototypes, yielding an intrinsic modality gap that cannot be eliminated by prompt engineering alone. To mitigate this gap under the post-hoc constraint, this paper presents an online pseudo-supervised framework that directly learns class prototypes in the visual feature space using unlabeled test-time data streams and soft predictions from the pre-trained VLMs. We provide theoretical guarantees for the convergence of the online optimization procedure. Extensive experiments empirically demonstrate that our method achieves a new state of the art across a variety of OOD detection setups.

Problem

• Out-of-distribution detection
• Vision-language models
• Modality gap
• Post-hoc detection
• Class prototypes

Innovation

• modality gap
• post-hoc OOD detection
• vision-language models
• online pseudo-supervised learning
• visual prototypes