Retrieval-Augmented Prompt for OOD Detection

2025-08-14

AI Summary

Existing out-of-distribution (OOD) detection methods rely on limited or mismatched outlier samples or in-distribution data, resulting in insufficient semantic supervision and poor generalization. To address this, we propose Retrieval-Enhanced Prompting (REPrompt), the first framework to integrate external textual knowledge retrieval into prompt-based OOD detection—enabling semantic enhancement during training and dynamic prompt adaptation at test time. Leveraging a pre-trained vision-language model, REPrompt retrieves semantically relevant descriptive terms from an external text corpus via cross-modal similarity matching, thereby constructing more discriminative OOD prompts. On the ImageNet-1k 1-shot OOD detection benchmark, REPrompt achieves state-of-the-art performance, reducing average false positive rate at 95% true positive rate (FPR95) by 7.05% and improving area under the receiver operating characteristic curve (AUROC) by 1.71% over prior methods. Our core contribution lies in the novel integration of external knowledge retrieval with dynamic prompting, substantially enhancing semantic awareness and adaptability to unseen classes.

Technology Category

• Computer Vision: Image and Video Retrieval
• Natural Language Processing: Prompt Engineering / Prompting
• Data Mining & Knowledge Management: Anomaly/Outlier Detection

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Out-of-Distribution (OOD) detection is crucial for the reliable deployment of machine learning models in-the-wild, enabling accurate identification of test samples that differ from the training data distribution. Existing methods rely on auxiliary outlier samples or in-distribution (ID) data to generate outlier information for training, but due to limited outliers and their mismatch with real test OOD samples, they often fail to provide sufficient semantic supervision, leading to suboptimal performance. To address this, we propose a novel OOD detection method called Retrieval-Augmented Prompt (RAP). RAP augments a pre-trained vision-language model's prompts by retrieving external knowledge, offering enhanced semantic supervision for OOD detection. During training, RAP retrieves descriptive words for outliers based on joint similarity with external textual knowledge and uses them to augment the model's OOD prompts. During testing, RAP dynamically updates OOD prompts in real-time based on the encountered OOD samples, enabling the model to rapidly adapt to the test environment. Our extensive experiments demonstrate that RAP achieves state-of-the-art performance on large-scale OOD detection benchmarks. For example, in 1-shot OOD detection on the ImageNet-1k dataset, RAP reduces the average FPR95 by 7.05% and improves the AUROC by 1.71% compared to previous methods. Additionally, comprehensive ablation studies validate the effectiveness of each module and the underlying motivations of our approach.

Problem

• Enhancing OOD detection with semantic supervision
• Addressing limited outlier data in training
• Improving real-time adaptation to test environments

Innovation

• Retrieval-Augmented Prompt enhances semantic supervision
• Dynamic real-time prompt updates for test samples
• Leverages external knowledge for outlier description