Harnessing Large Language and Vision-Language Models for Robust Out-of-Distribution Detection

2025-01-09

AI Summary

Existing zero-shot out-of-distribution (OOD) detection methods suffer from suboptimal performance in both near- and far-OOD settings, particularly lacking robustness against OOD samples semantically confusable with in-distribution (ID) classes. To address this, we propose an LLM–VLM collaborative framework: a large language model (LLM) generates superclass and background descriptions for ID classes; CLIP is leveraged to disentangle semantic and background features, followed by background subtraction to extract discriminative core representations; and a dual-path prompt tuning strategy—combining few-shot textual prompts and visual prompts—is introduced to enhance domain adaptability. This work is the first to jointly optimize near- and far-OOD detection within a unified LLM–VLM framework. Extensive experiments demonstrate substantial improvements over state-of-the-art methods across multiple benchmarks, achieving up to +2.9% AUROC and −12.6% FPR95, while significantly improving robustness under cross-domain covariate shift.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Out-of-distribution (OOD) detection has seen significant advancements with zero-shot approaches by leveraging the powerful Vision-Language Models (VLMs) such as CLIP. However, prior research works have predominantly focused on enhancing Far-OOD performance, while potentially compromising Near-OOD efficacy, as observed from our pilot study. To address this issue, we propose a novel strategy to enhance zero-shot OOD detection performances for both Far-OOD and Near-OOD scenarios by innovatively harnessing Large Language Models (LLMs) and VLMs. Our approach first exploit an LLM to generate superclasses of the ID labels and their corresponding background descriptions followed by feature extraction using CLIP. We then isolate the core semantic features for ID data by subtracting background features from the superclass features. The refined representation facilitates the selection of more appropriate negative labels for OOD data from a comprehensive candidate label set of WordNet, thereby enhancing the performance of zero-shot OOD detection in both scenarios. Furthermore, we introduce novel few-shot prompt tuning and visual prompt tuning to adapt the proposed framework to better align with the target distribution. Experimental results demonstrate that the proposed approach consistently outperforms current state-of-the-art methods across multiple benchmarks, with an improvement of up to 2.9% in AUROC and a reduction of up to 12.6% in FPR95. Additionally, our method exhibits superior robustness against covariate shift across different domains, further highlighting its effectiveness in real-world scenarios.

Problem

• Zero-shot OOD Detection
• Large Language Models
• Visual Language Models

Innovation

• Large Language Models
• Visual Language Models
• CLIP Feature Extraction