EditAR: Unified Conditional Generation with Autoregressive Models

2025-01-08

AI Summary

Existing diffusion models struggle to uniformly support diverse image generation and editing tasks. To address this, we propose EditAR—the first fully unified autoregressive framework for multimodal image synthesis and editing. EditAR jointly processes images and natural language instructions as input, tokenizes heterogeneous conditioning signals—including depth maps, edge maps, and segmentation masks—into a shared discrete space, and generates image sequences via standard next-token prediction. Its key contributions are: (1) the first fully unified autoregressive formulation for conditional image generation across multiple tasks; and (2) a text–image alignment enhancement mechanism leveraging knowledge distillation from large language models. Evaluated on multiple benchmarks, EditAR matches or approaches task-specific state-of-the-art methods in generation quality, fidelity, and controllability, demonstrating the effectiveness and strong generalization capability of the unified autoregressive paradigm for controllable image synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Recent progress in controllable image generation and editing is largely driven by diffusion-based methods. Although diffusion models perform exceptionally well in specific tasks with tailored designs, establishing a unified model is still challenging. In contrast, autoregressive models inherently feature a unified tokenized representation, which simplifies the creation of a single foundational model for various tasks. In this work, we propose EditAR, a single unified autoregressive framework for a variety of conditional image generation tasks, e.g., image editing, depth-to-image, edge-to-image, segmentation-to-image. The model takes both images and instructions as inputs, and predicts the edited images tokens in a vanilla next-token paradigm. To enhance the text-to-image alignment, we further propose to distill the knowledge from foundation models into the autoregressive modeling process. We evaluate its effectiveness across diverse tasks on established benchmarks, showing competitive performance to various state-of-the-art task-specific methods. Project page: https://jitengmu.github.io/EditAR/

Problem

• Image Generation
• Adaptive Multitasking
• Autoregressive Models

Innovation

• EditAR
• Unified Image Generation
• Autoregressive Model