SimLabel: Consistency-Guided OOD Detection with Pretrained Vision-Language Models

2025-01-20

AI Summary

Existing out-of-distribution (OOD) detection methods predominantly rely on either single-class or full-class in-distribution (ID) labels, neglecting semantic relationships among normal classes—thereby limiting the OOD discrimination capability of vision-language models (VLMs) in zero-shot settings. To address this, we propose SimLabel: the first method leveraging consistency in semantic similarity among ID classes to construct a consistency-guided image–class similarity metric; and a post-hoc multi-label weighting strategy that overcomes the limitations of conventional single-label or full-set modeling. Built upon pre-trained VLMs (e.g., CLIP), SimLabel integrates class label semantic embeddings with consistency-weighted similarity computation. Extensive experiments on multiple zero-shot OOD benchmarks demonstrate significant improvements over state-of-the-art methods, with strong generalization across diverse datasets, compatibility with various VLM backbones, and publicly released code and interactive demo.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Detecting out-of-distribution (OOD) data is crucial in real-world machine learning applications, particularly in safety-critical domains. Existing methods often leverage language information from vision-language models (VLMs) to enhance OOD detection by improving confidence estimation through rich class-wise text information. However, when building OOD detection score upon on in-distribution (ID) text-image affinity, existing works either focus on each ID class or whole ID label sets, overlooking inherent ID classes' connection. We find that the semantic information across different ID classes is beneficial for effective OOD detection. We thus investigate the ability of image-text comprehension among different semantic-related ID labels in VLMs and propose a novel post-hoc strategy called SimLabel. SimLabel enhances the separability between ID and OOD samples by establishing a more robust image-class similarity metric that considers consistency over a set of similar class labels. Extensive experiments demonstrate the superior performance of SimLabel on various zero-shot OOD detection benchmarks. The proposed model is also extended to various VLM-backbones, demonstrating its good generalization ability. Our demonstration and implementation codes are available at: https://github.com/ShuZou-1/SimLabel.

Problem

• Out-of-Distribution (OOD) Detection
• Inter-Class Correlation
• Anomaly Detection Performance

Innovation

• SimLabel
• OOD Detection
• Inter-Class Similarity