Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models

2025-04-24

AI Summary

Autoregressive (AR) image generation suffers from prohibitive computational overhead at high resolutions due to the excessive number of image tokens, severely limiting training and inference efficiency. To address this, we propose a token-shuffle/unshuffle mechanism that locally merges image tokens along the channel dimension to drastically compress sequence length during training, followed by spatial decoupling for reconstruction during inference—requiring no additional pretraining or separate text encoder and enabling end-to-end joint optimization. This work achieves, for the first time, pure AR text-to-image generation at 2048×2048 resolution. Leveraging redundancy between low-dimensional visual codes and high-dimensional linguistic vocabularies, we design a lightweight, unified next-token prediction architecture. On GenAI-benchmark, our 2.7B-parameter model scores 0.77 on challenging prompts—surpassing LlamaGen (AR) by 0.18 and LDM (diffusion) by 0.15. Large-scale human evaluation confirms superior performance in text alignment, visual fidelity, and perceptual quality.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Autoregressive (AR) models, long dominant in language generation, are increasingly applied to image synthesis but are often considered less competitive than Diffusion-based models. A primary limitation is the substantial number of image tokens required for AR models, which constrains both training and inference efficiency, as well as image resolution. To address this, we present Token-Shuffle, a novel yet simple method that reduces the number of image tokens in Transformer. Our key insight is the dimensional redundancy of visual vocabularies in Multimodal Large Language Models (MLLMs), where low-dimensional visual codes from visual encoder are directly mapped to high-dimensional language vocabularies. Leveraging this, we consider two key operations: token-shuffle, which merges spatially local tokens along channel dimension to decrease the input token number, and token-unshuffle, which untangles the inferred tokens after Transformer blocks to restore the spatial arrangement for output. Jointly training with textual prompts, our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis in a unified next-token prediction way while maintaining efficient training and inference. For the first time, we push the boundary of AR text-to-image generation to a resolution of 2048x2048 with gratifying generation performance. In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15. Exhaustive large-scale human evaluations also demonstrate our prominent image generation ability in terms of text-alignment, visual flaw, and visual appearance. We hope that Token-Shuffle can serve as a foundational design for efficient high-resolution image generation within MLLMs.

Problem

• Reduces image tokens to enhance AR model efficiency
• Enables high-resolution image synthesis up to 2048x2048
• Improves text-to-image alignment and visual quality

Innovation

• Token-Shuffle reduces image tokens in Transformer
• Merges local tokens along channel dimension
• Enables 2048x2048 resolution image generation