Vague2Detect: Handling Ambiguous Prompts in Knowledge-Based Open-World Detection

2026-09-09

AI Summary

为解决YOLO等模型处理模糊提示的局限性，提出Vague2Detect方法，结合Sentence-BERT、知识库及GPT-3.5-turbo生成候选描述，显著提高模糊提示的成功检测率。

Abstract

Real-world detectors must often interpret functional or ambiguous prompts, yet conventional models such as YOLO remain restricted to fixed class lists. Even open-vocabulary models like YOLO-World frequently misalign vague language with the intended objects. Building on our prior work Commonsense-Guided Open-World Object Detection Using LLMs and Visual-Semantic Matching, we address YOLO-World's limitations in grounding task-driven queries. We propose Vague2Detect, a hybrid pipeline in which a fine-tuned Sentence-BERT retrieves candidates from a structured household Knowledge Base (KB), and YOLO-World verifies their presence in the image. For prompts outside the KB, a large language model (GPT-3.5-turbo) generates candidate descriptions, dynamically expanding the KB to cover novel concepts. On a benchmark of household scenes using custom images and an Open Images V7 subset, YOLO-World alone achieves only 32% Vague Prompt Success Rate (VPSR), the ability to map ambiguous queries to correct detections. In contrast, Vague2Detect improves performance to 61% VPSR with high precision, and up to 85% when augmented with GPT fallback.

Problem

• ambiguous prompts
• open-vocabulary models
• knowledge-based detection

Innovation

• Vague2Detect
• Sentence-BERT
• Knowledge Base Expansion
• Open-Vocabulary Detection
• GPT-3.5-turbo