HoloCount: A Holistic Visual Counting Benchmark for MLLMs

2026-07-07

AI Summary

This work addresses the susceptibility of multimodal large language models (MLLMs) to numerical hallucinations in visual counting tasks and the inadequacy of existing benchmarks in evaluating their quantitative reasoning under complex logical and adversarial conditions. The authors propose the first comprehensive visual counting benchmark tailored for MLLMs, featuring a three-tier evaluation framework encompassing semantic counting, analytical counting, and robustness testing. This benchmark introduces advanced challenges such as logical composition, set-based reasoning, and anti-prior interference. Large-scale evaluations across more than twenty state-of-the-art models reveal that while current MLLMs can handle basic counting, their performance degrades significantly when fine-grained localization, spatial reasoning, or resistance to linguistic biases and high-density distractors is required. These findings expose a critical capability gap between perception and complex reasoning, offering clear directions for future model improvement.

Abstract

Visual counting is a fundamental pillar of multimodal intelligence, requiring a seamless integration of fine-grained grounding and spatial reasoning. While Multimodal Large Language Models (MLLMs) have achieved remarkable success in qualitative scene understanding, their quantitative precision remains a significant bottleneck, often characterized by persistent numerical hallucinations. Existing counting benchmarks primarily focus on basic perception in simplified contexts, failing to capture the complex failure modes that emerge under logical constraints or adversarial conditions. To address these limitations, we introduce HoloCount, a holistic and diagnostically rich benchmark structured around a three-level hierarchical taxonomy. HoloCount evaluates MLLMs across: (1) Semantic Counting, focusing on atomic and property-based enumeration; (2) Analytical Counting, assessing logical composition through spatial and set-based reasoning; and (3) Robustness Testing, probing model integrity against adverse scenarios and grounded counter-priors, such as high-density scenes and linguistic biases. Through an exhaustive evaluation of over 20 state-of-the-art MLLMs, we reveal a critical performance gap: even top-tier models degrade significantly as tasks transition from perception to complex analytical reasoning and adverse scenarios. Our findings provide a systematic landscape of current MLLM counting capabilities and offer a roadmap for developing more grounded and reliable multimodal systems. The dataset is available at https://mm-mvr.github.io/HoloCount/.

Problem

• visual counting
• multimodal large language models
• numerical hallucinations
• robustness
• spatial reasoning

Innovation

• visual counting
• multimodal large language models
• holistic benchmark
• analytical reasoning
• robustness testing