CountGD++: Generalized Prompting for Open-World Counting

2025-12-29

AI Summary

Existing open-world object counting methods are constrained by manual visual exemplar selection via bounding boxes, which hinders robust foreground-background separation and lacks automated annotation mechanisms. To address these limitations, we propose the first flexible counting framework supporting *joint positive–negative prompting*: it jointly specifies *what to count* and *what to exclude* using multimodal cues—textual descriptions and visual exemplars (including explicit negative examples)—eliminating the need for manual annotations. Our contributions include: (i) the first negative-prompting mechanism for counting; (ii) a novel cross-domain (natural + synthetic) pseudo-exemplar generation technique; and (iii) integration of the counting model as a multimodal visual expert within a large language model (LLM), forming an LLM-vision agent architecture. Extensive experiments on multiple open-world counting benchmarks demonstrate significant improvements in accuracy, cross-domain generalization, and inference efficiency. The code is publicly available.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The flexibility and accuracy of methods for automatically counting objects in images and videos are limited by the way the object can be specified. While existing methods allow users to describe the target object with text and visual examples, the visual examples must be manually annotated inside the image, and there is no way to specify what not to count. To address these gaps, we introduce novel capabilities that expand how the target object can be specified. Specifically, we extend the prompt to enable what not to count to be described with text and/or visual examples, introduce the concept of `pseudo-exemplars' that automate the annotation of visual examples at inference, and extend counting models to accept visual examples from both natural and synthetic external images. We also use our new counting model, CountGD++, as a vision expert agent for an LLM. Together, these contributions expand the prompt flexibility of multi-modal open-world counting and lead to significant improvements in accuracy, efficiency, and generalization across multiple datasets. Code is available at https://github.com/niki-amini-naieni/CountGDPlusPlus.

Problem

• Enables specifying what not to count via text or visual examples
• Automates annotation of visual examples using pseudo-exemplars at inference
• Extends counting models to accept natural and synthetic external images

Innovation

• Extends prompts to exclude objects using text or visual examples
• Introduces pseudo-exemplars to automate visual example annotation
• Accepts visual examples from both natural and synthetic external images