CountCluster: Training-Free Object Quantity Guidance with Cross-Attention Map Clustering for Text-to-Image Generation

2025-08-14

AI Summary

To address inaccurate object count control in diffusion-based text-to-image generation, this work identifies that object quantity is determined early in the denoising process. Building on this insight, we propose a lightweight, training-free guidance method that requires no external modules: it applies k-means clustering to cross-attention maps from initial denoising steps to partition the spatial layout; then jointly optimizes attention distributions—via attention-score–based region masking and latent-space fine-tuning—to align with the target count specified in the prompt. The method preserves image fidelity and diversity while achieving an average 18.5% improvement in object counting accuracy across multiple benchmark prompts. It significantly outperforms existing count-control techniques and establishes a novel paradigm for controllable generation in diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Diffusion-based text-to-image generation models have demonstrated strong performance in terms of image quality and diversity. However, they still struggle to generate images that accurately reflect the number of objects specified in the input prompt. Several approaches have been proposed that rely on either external counting modules for iterative refinement or quantity representations derived from learned tokens or latent features. However, they still have limitations in accurately reflecting the specified number of objects and overlook an important structural characteristic--The number of object instances in the generated image is largely determined in the early timesteps of the denoising process. To correctly reflect the object quantity for image generation, the highly activated regions in the object cross-attention map at the early timesteps should match the input object quantity, while each region should be clearly separated. To address this issue, we propose extit{CountCluster}, a method that guides the object cross-attention map to be clustered according to the specified object count in the input, without relying on any external tools or additional training. The proposed method partitions the object cross-attention map into $k$ clusters at inference time based on attention scores, defines an ideal distribution in which each cluster is spatially well-separated, and optimizes the latent to align with this target distribution. Our method achieves an average improvement of 18.5%p in object count accuracy compared to existing methods, and demonstrates superior quantity control performance across a variety of prompts. Code will be released at: https://github.com/JoohyeonL22/CountCluster .

Problem

• Accurately generating specified object counts in text-to-image models
• Aligning cross-attention map clusters with target object quantity
• Improving object count accuracy without external tools or training

Innovation

• Clusters cross-attention maps by object count
• Optimizes latents for spatially separated clusters
• Training-free guidance using attention score partitioning