Demystifying Numerosity in Diffusion Models -- Limitations and Remedies

2025-10-13

AI Summary

Diffusion models struggle to accurately adhere to object cardinality instructions in text-to-image generation. This work identifies the root cause as an implicit cardinality bias embedded in the noise prior—not insufficient data or model scale. To rigorously diagnose this issue, we introduce two fine-grained counting benchmarks—GrayCount250 and NaturalCount6—and conduct controlled ablation studies to empirically validate the mechanism. Building on this insight, we propose a counting-aware layout guidance strategy that explicitly injects target cardinality information during noise initialization, thereby modulating the diffusion process at the prior level. Our method requires no increase in model size or training data. It boosts counting accuracy from 20.0% to 85.3% on GrayCount250 and from 74.8% to 86.3% on NaturalCount6, substantially surpassing the performance ceiling of conventional scaling-based approaches.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Numerosity remains a challenge for state-of-the-art text-to-image generation models like FLUX and GPT-4o, which often fail to accurately follow counting instructions in text prompts. In this paper, we aim to study a fundamental yet often overlooked question: Can diffusion models inherently generate the correct number of objects specified by a textual prompt simply by scaling up the dataset and model size? To enable rigorous and reproducible evaluation, we construct a clean synthetic numerosity benchmark comprising two complementary datasets: GrayCount250 for controlled scaling studies, and NaturalCount6 featuring complex naturalistic scenes. Second, we empirically show that the scaling hypothesis does not hold: larger models and datasets alone fail to improve counting accuracy on our benchmark. Our analysis identifies a key reason: diffusion models tend to rely heavily on the noise initialization rather than the explicit numerosity specified in the prompt. We observe that noise priors exhibit biases toward specific object counts. In addition, we propose an effective strategy for controlling numerosity by injecting count-aware layout information into the noise prior. Our method achieves significant gains, improving accuracy on GrayCount250 from 20.0% to 85.3% and on NaturalCount6 from 74.8% to 86.3%, demonstrating effective generalization across settings.

Problem

• Diffusion models fail to generate correct object counts from text prompts
• Scaling model size and dataset does not improve counting accuracy
• Noise initialization biases prevent models from following numerosity instructions

Innovation

• Injecting count-aware layout into noise prior
• Addressing numerosity bias in diffusion models
• Improving counting accuracy via layout control