Mitigating the Modality Gap: Few-Shot Out-of-Distribution Detection with Multi-modal Prototypes and Image Bias Estimation

2025-02-02

AI Summary

To address the high false-positive rate in vision-language models (VLMs) for anomaly detection—stemming from modality misalignment between images and text—this paper proposes SUPREME, a training-free prototype enhancement method coupled with a few-shot fine-tuning framework. Its core contributions are: (1) the first multimodal prototype co-learning mechanism, jointly leveraging in-distribution (ID) image and text prototypes; (2) bias-guided prompt generation (BPG), grounded in Gaussian estimation of image-level deviations, and image-text consistency optimization (ITC); and (3) a cross-modal out-of-distribution (OOD) scoring metric, S_GMP. Evaluated across multiple benchmarks, SUPREME consistently achieves lower false-positive rates than existing VLM-based OOD detection methods. Notably, it delivers performance gains even without any additional training, demonstrating both efficacy and practicality.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Existing vision-language model (VLM)-based methods for out-of-distribution (OOD) detection typically rely on similarity scores between input images and in-distribution (ID) text prototypes. However, the modality gap between image and text often results in high false positive rates, as OOD samples can exhibit high similarity to ID text prototypes. To mitigate the impact of this modality gap, we propose incorporating ID image prototypes along with ID text prototypes. We present theoretical analysis and empirical evidence indicating that this approach enhances VLM-based OOD detection performance without any additional training. To further reduce the gap between image and text, we introduce a novel few-shot tuning framework, SUPREME, comprising biased prompts generation (BPG) and image-text consistency (ITC) modules. BPG enhances image-text fusion and improves generalization by conditioning ID text prototypes on the Gaussian-based estimated image domain bias; ITC reduces the modality gap by minimizing intra- and inter-modal distances. Moreover, inspired by our theoretical and empirical findings, we introduce a novel OOD score $S_{ extit{GMP}}$, leveraging uni- and cross-modal similarities. Finally, we present extensive experiments to demonstrate that SUPREME consistently outperforms existing VLM-based OOD detection methods.

Problem

• Visual Language Model
• Anomaly Detection
• Modality Discrepancy

Innovation

• SUPREME method
• modal discrepancy
• anomaly detection