CounterCount: A Diagnostic Framework for Counting Bias in Vision Language Models

2026-05-18

AI Summary

This work addresses the susceptibility of vision-language models to object-level linguistic priors in counting tasks, which often leads them to overlook genuine visual evidence—particularly when visual input contradicts commonsense expectations. To systematically diagnose this overreliance on priors, the authors introduce CounterCount, a diagnostic framework that constructs factual–counterfactual image pairs through controlled image editing, complemented by localized evidence annotations and attention analysis. Building on these insights, they propose an inference-time attention modulation strategy that dynamically reweights visual tokens to enhance the model’s focus on counting-relevant visual features. Experiments across multiple state-of-the-art models demonstrate that this approach improves counterfactual counting accuracy by up to 8%, substantially increasing the reliability of visual grounding in numerical reasoning.

Abstract

Vision-Language Models (VLMs) excel at multimodal reasoning, yet it remains unclear whether their answers are grounded in visual evidence or driven by learned language and world priors. Counting provides a precise testbed: when visual evidence conflicts with canonical object knowledge, a model must rely on the image rather than a prototypical count. We introduce CounterCount, a diagnostic framework for counterfactual counting in VLMs, consisting of paired factual and counterfactual images with edited count-relevant attributes, verified answers, and localized evidence annotations. Evaluating recent VLMs, we find strong performance on factual images but consistent degradation under counterfactual attribute changes, indicating reliance on object-level priors even when contradictory visual evidence is present. Using localized annotations, we show that these failures are not solely due to missing or ambiguous visual evidence, but to models underweighting attention to count-relevant visual tokens. We introduce a unified inference-time attention modulation strategy that reweights selected visual tokens, improving counterfactual counting accuracy by up to 8% across multiple VLMs. Overall, CounterCount exposes prior-driven counting failures and provides diagnostic insights for designing future VLMs.

Problem

• counting bias
• vision-language models
• visual grounding
• counterfactual reasoning
• object priors

Innovation

• counterfactual counting
• vision-language models
• attention modulation
• diagnostic framework
• visual grounding