Improving Sample Diversity in Autoregressive Text-to-Image Generation via Cluster Truncation

2026-07-11

AI Summary

This work addresses the limited sample diversity in autoregressive text-to-image (T2I) generation, which stems from the inefficacy of existing token-level decoding strategies due to high entropy and visual token redundancy. The study presents the first systematic analysis of this diversity bottleneck and introduces an innovative $p$-less cluster decoding strategy. By clustering visually similar tokens and performing entropy-driven truncation sampling at the cluster level rather than the individual token level, the method elevates diversity enhancement from token-level to cluster-level. Extensive experiments across four autoregressive T2I models and two datasets demonstrate that the proposed approach significantly improves sample diversity while preserving image quality and alignment with input prompts.

Abstract

While diffusion models achieve state-of-the-art image quality for text-to-image (T2I) generation, recent work has demonstrated that they suffer from sample diversity collapse. In this work, we investigate whether autoregressive (AR) image generation models can push the Pareto frontier between image quality and sample diversity. With recent advances in quality and efficiency, AR models have emerged as a viable alternative to diffusion-based image generation. Beyond enabling new use cases such as interleaved image-text generation, their sequential generation process makes them compatible with a wide range of token-based decoding strategies originally developed to improve diversity in text generation. Motivated by the potential of a better diversity-quality tradeoff in the AR paradigm, we present the first systematic study of sample diversity in AR image generation models. We show that two key properties of AR image generation, persistently high token-level entropy and substantial redundancy in visual token spaces, limit the effectiveness of existing token-level decoding methods for diversity enhancement. We therefore propose $p$-less cluster, a new decoding strategy that performs entropy-based truncation sampling at cluster level rather than at token level. We evaluate our approach and baseline decoding methods across four autoregressive T2I models and two datasets using a comprehensive suite of metrics spanning image quality, prompt alignment, and diversity. Our results show that $p$-less cluster unlocks the greatest diversity across most evaluated autoregressive T2I models and datasets while maintaining image quality and prompt alignment.

Problem

• sample diversity
• autoregressive text-to-image generation
• token-level decoding
• visual token redundancy
• diversity-quality tradeoff

Innovation

• autoregressive image generation
• sample diversity
• cluster truncation
• token-level entropy
• decoding strategy