NegAS: Negative Label Guided Attention and Scoring for Out-of-Distribution Object Detection with Vision-Language Models

2026-06-21

AI Summary

This work addresses the limited out-of-distribution (OOD) detection capability of vision-language models (VLMs) in object detection, particularly tackling two key challenges: the underutilization of background regions and the incompatibility between multi-label outputs and OOD scoring. To this end, the authors propose NegAS, a novel framework that introduces, for the first time, a negative-label guidance mechanism into VLM-based detectors. Leveraging a large language model to generate semantically related yet class-distinct negative labels, NegAS incorporates a NegA attention module to highlight potential OOD regions and devises a sigmoid-based NegS scoring function that aligns with multi-label prediction for effective OOD discrimination. Evaluated on COCO and OpenImages, NegAS reduces FPR95 by 11.4% and 25.5%, respectively, substantially improving OOD detection performance while preserving in-distribution detection accuracy.

Abstract

Out-of-Distribution (OOD) detection is essential for ensuring the robustness and reliability of object detection systems deployed in safety-critical applications. While prior research has mainly focused on uni-modal detectors or vision-language model (VLM) based classifiers, the potential of VLM-based object detectors in OOD scenarios remains underexplored. In this work, we take the first step toward building OOD object detection methods upon VLMs. We identify two challenges specific to VLM detectors: (i) their text-guided attention enhances foreground with ID labels but treats background uniformly, leaving potential OOD regions unexploited for separating in-distribution (ID) from OOD instances; and (ii) their sigmoid-based multi-label outputs are incompatible with softmax-based OOD scores, calling for scoring functions consistent with VLM probabilistic outputs. Hence, we introduce Negative Label Guided Attention and Scoring (NegAS). To address (i), we propose a negative label guided attention module (NegA), where LLM-generated, visually-similar but semantically-different negative labels are used to guide attention toward potential OOD background regions. To address (ii), we introduce a novel sigmoid-based OOD scoring function (NegS) that leverages both ID and negative labels, producing strong responses for ID instances and suppressed responses for OOD ones. Extensive experiments demonstrate that our approach improves OOD detection performance by a large margin while maintaining ID accuracy, e.g., reducing the FPR95 by 11.4% on the COCO dataset and 25.5% on the OpenImages dataset compared to the baseline model. While initially designed for dense VLM detectors like YOLO-World, we successfully adapt NegAS to Grounding DINO, a query-based VLM transformer and achieve significant improvements, demonstrating the generalizability of our framework.

Problem

• Out-of-Distribution Detection
• Vision-Language Models
• Object Detection
• Attention Mechanism
• OOD Scoring

Innovation

• Negative Label Guidance
• Vision-Language Models
• Out-of-Distribution Detection
• Attention Mechanism
• Sigmoid-based Scoring