ControlAR: Controllable Image Generation with Autoregressive Models

2024-10-03

arXiv.org

AI Summary

To address the challenge of efficiently integrating spatial controls—such as edges, depth maps, and segmentation masks—into autoregressive (AR) image generation, this paper proposes the first conditional decoding-based controllable AR framework supporting arbitrary resolutions. Methodologically, we design a lightweight control encoder and a token-wise control–image feature fusion mechanism, abandoning the inefficient paradigm of control token pre-filling. We further introduce a spatial control tokenization strategy to ensure high-fidelity alignment between control signals and generated images. Extensive experiments demonstrate that our approach consistently outperforms state-of-the-art controllable diffusion models—including ControlNet++—across diverse control inputs: it achieves new SOTA in both FID and CLIP-Score, delivers superior visual quality, and supports dynamic resolution output without architectural modification.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Autoregressive (AR) models have reformulated image generation as next-token prediction, demonstrating remarkable potential and emerging as strong competitors to diffusion models. However, control-to-image generation, akin to ControlNet, remains largely unexplored within AR models. Although a natural approach, inspired by advancements in Large Language Models, is to tokenize control images into tokens and prefill them into the autoregressive model before decoding image tokens, it still falls short in generation quality compared to ControlNet and suffers from inefficiency. To this end, we introduce ControlAR, an efficient and effective framework for integrating spatial controls into autoregressive image generation models. Firstly, we explore control encoding for AR models and propose a lightweight control encoder to transform spatial inputs (e.g., canny edges or depth maps) into control tokens. Then ControlAR exploits the conditional decoding method to generate the next image token conditioned on the per-token fusion between control and image tokens, similar to positional encodings. Compared to prefilling tokens, using conditional decoding significantly strengthens the control capability of AR models but also maintains the model's efficiency. Furthermore, the proposed ControlAR surprisingly empowers AR models with arbitrary-resolution image generation via conditional decoding and specific controls. Extensive experiments can demonstrate the controllability of the proposed ControlAR for the autoregressive control-to-image generation across diverse inputs, including edges, depths, and segmentation masks. Furthermore, both quantitative and qualitative results indicate that ControlAR surpasses previous state-of-the-art controllable diffusion models, e.g., ControlNet++. Code, models, and demo will soon be available at https://github.com/hustvl/ControlAR.

Problem

• Image Generation
• Autoregressive Models
• ControlNet

Innovation

• ControlAR
• Guided Image Generation
• Controllable Diffusion Models