🤖 AI Summary
Text-to-image diffusion models frequently exhibit object counting errors when responding to quantitative prompts (e.g., “five apples and ten lemons”). To address this, we propose a reference-free, category-agnostic counting network with gradient-guided denoising— the first method to backpropagate a differentiable counting loss through attention maps into the diffusion denoising process. Our approach enables independent mask-based segmentation and per-object denoising control for multiple categories. It integrates three key components: noise-prediction fine-tuning, attention-driven fine-grained segmentation, and gradient-guided denoising optimization. Crucially, the method preserves FID stability while achieving an average counting accuracy improvement of over 40% relative to baseline models. This substantially enhances high-fidelity generation under complex, multi-class quantitative prompts, demonstrating significant advances in controllable, numerically accurate image synthesis.
📝 Abstract
Recently, there have been significant improvements in the quality and performance of text-to-image generation, largely due to the impressive results attained by diffusion models. However, text-to-image diffusion models sometimes struggle to create high-fidelity content for the given input prompt. One specific issue is their difficulty in generating the precise number of objects specified in the text prompt. For example, when provided with the prompt"five apples and ten lemons on a table,"images generated by diffusion models often contain an incorrect number of objects. In this paper, we present a method to improve diffusion models so that they accurately produce the correct object count based on the input prompt. We adopt a counting network that performs reference-less class-agnostic counting for any given image. We calculate the gradients of the counting network and refine the predicted noise for each step. To address the presence of multiple types of objects in the prompt, we utilize novel attention map guidance to obtain high-quality masks for each object. Finally, we guide the denoising process using the calculated gradients for each object. Through extensive experiments and evaluation, we demonstrate that the proposed method significantly enhances the fidelity of diffusion models with respect to object count. Code is available at https://github.com/furiosa-ai/counting-guidance.
