Zero-shot Object-Level OOD Detection with Context-Aware Inpainting

2024-02-05

arXiv.org

AI Summary

Existing object detectors suffer from overconfidence on out-of-distribution (OOD) objects, compromising deployment reliability. This paper addresses the black-box model setting and proposes the first zero-shot, object-level OOD detection method leveraging pre-trained diffusion models. Our approach guides a diffusion model—using only detected bounding boxes and class labels—to perform context-aware, conditional image inpainting. Discrimination arises from the fidelity gap: in-distribution (ID) objects are reconstructed with high fidelity, whereas OOD objects exhibit pronounced reconstruction distortions. Crucially, our method requires no training data, fine-tuning, or access to ID samples. It is compatible with off-the-shelf diffusion models (e.g., Stable Diffusion) and achieves state-of-the-art performance under zero-shot settings across multiple benchmarks. Moreover, it remains competitive against supervised baselines in comparative experiments, significantly enhancing robustness for fine-grained, object-level OOD identification.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Machine learning algorithms are increasingly provided as black-box cloud services or pre-trained models, without access to their training data. This motivates the problem of zero-shot out-of-distribution (OOD) detection. Concretely, we aim to detect OOD objects that do not belong to the classifier's label set but are erroneously classified as in-distribution (ID) objects. Our approach, RONIN, uses an off-the-shelf diffusion model to replace detected objects with inpainting. RONIN conditions the inpainting process with the predicted ID label, drawing the input object closer to the in-distribution domain. As a result, the reconstructed object is very close to the original in the ID cases and far in the OOD cases, allowing RONIN to effectively distinguish ID and OOD samples. Throughout extensive experiments, we demonstrate that RONIN achieves competitive results compared to previous approaches across several datasets, both in zero-shot and non-zero-shot settings.

Problem

• Detecting novel objects not seen during training
• Overcoming overconfidence in modern object detectors
• Using generative models to identify out-of-distribution objects

Innovation

• Uses class-conditioned inpainting for OOD detection
• Leverages text-to-image generative model inconsistencies
• Measures reconstruction error as OOD indicator