LLM-Guided Agentic Object Detection for Open-World Understanding

2025-07-14

AI Summary

Traditional object detection is constrained by predefined categories, limiting generalization to unknown objects. While open-world object detection (OWOD) and open-vocabulary object detection (OVOD) improve flexibility, OWOD lacks semantic labels for unknown classes, and OVOD relies on handcrafted prompts, compromising autonomy. This paper proposes LAOD, the first framework to decouple object localization from semantic naming: a large language model (LLM) autonomously generates scene-aware zero-shot class names, which are then used by an open-vocabulary detector for localization. To evaluate this paradigm, we introduce two novel metrics—Classification-Aware Average Precision (CAAP) for localization accuracy and Semantic Naming Average Precision (SNAP) for naming fidelity. Experiments on LVIS, COCO, and COCO-OOD demonstrate that LAOD significantly improves end-to-end detection and interpretable naming of unknown objects, enhancing autonomous adaptability in open-world environments.

Technology Category

• Computer Vision: Language and Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Object detection traditionally relies on fixed category sets, requiring costly re-training to handle novel objects. While Open-World and Open-Vocabulary Object Detection (OWOD and OVOD) improve flexibility, OWOD lacks semantic labels for unknowns, and OVOD depends on user prompts, limiting autonomy. We propose an LLM-guided agentic object detection (LAOD) framework that enables fully label-free, zero-shot detection by prompting a Large Language Model (LLM) to generate scene-specific object names. These are passed to an open-vocabulary detector for localization, allowing the system to adapt its goals dynamically. We introduce two new metrics, Class-Agnostic Average Precision (CAAP) and Semantic Naming Average Precision (SNAP), to separately evaluate localization and naming. Experiments on LVIS, COCO, and COCO-OOD validate our approach, showing strong performance in detecting and naming novel objects. Our method offers enhanced autonomy and adaptability for open-world understanding.

Problem

• Enables label-free zero-shot object detection using LLMs
• Overcomes limitations of fixed category sets in traditional detection
• Introduces new metrics to evaluate localization and naming separately

Innovation

• LLM-guided label-free zero-shot object detection
• Dynamic scene-specific object name generation
• New metrics CAAP and SNAP for evaluation