Does it Really Count? Assessing Semantic Grounding in Text-Guided Class-Agnostic Counting

2026-05-04

AI Summary

This work addresses a critical limitation in existing text-guided, category-agnostic counting methods: their poor semantic grounding ability, which undermines reliable alignment between natural language prompts and target visual objects in real-world scenarios. To tackle this issue, the authors introduce PrACo++, a novel evaluation suite incorporating negative labels and distractor-based tests, along with MUCCA, a multi-category annotated dataset, thereby establishing the first semantic grounding evaluation framework tailored for multi-category counting. They further propose dedicated metrics and an evaluation protocol that integrate semantic similarity analysis to diagnose model failure modes. Experiments on ten state-of-the-art methods reveal that, despite strong performance on standard counting metrics, these models exhibit significant deficiencies in semantic alignment, underscoring the effectiveness and necessity of the proposed framework.

Abstract

Open-world text-guided class-agnostic counting (CAC) has emerged as a flexible paradigm for counting arbitrary object classes by using natural language prompts. However, current evaluation protocols primarily focus on standard counting errors within single-category images, overlooking a fundamental requirement: the ability to correctly ground the textual prompt in the visual scene. In this paper, we show that several state-of-the-art CAC models often struggle to determine which object class should be counted based on the given prompt, revealing a misalignment between textual semantics and visual object representations. This limitation leads to spurious counting responses and reduced reliability in real-world scenarios. To systematically address these limitations, we propose a new evaluation framework focused on model robustness and trustworthiness. Our contribution is two-fold: (i) we introduce PrACo++ (Prompt-Aware Counting++), a novel test suite featuring two dedicated evaluation protocols -- the negative-label test and the distractor test -- paired with new specialized metrics; and (ii) we present the MUCCA (MUlti-Category Class-Agnostic counting) evaluation dataset, a new collection of real-world images featuring multiple annotated object categories per scene, unlike existing CAC benchmarks that typically include a single category per image. Our extensive experimental evaluation of 10 state-of-the-art methods shows that, despite strong performance under standard counting metrics, current models exhibit significant weaknesses in understanding and grounding object class descriptions. Finally, we provide a quantitative analysis of how semantic similarity between prompts influences these failures. Overall, our results underscore the need for more semantically grounded architectures and offer a reliable framework for future assessment in open-world text-guided CAC methods.

Problem

• semantic grounding
• class-agnostic counting
• text-guided counting
• open-world counting
• visual-text alignment

Innovation

• semantic grounding
• class-agnostic counting
• text-guided counting
• evaluation framework
• multi-category dataset