End-to-End Autoregressive Image Generation with 1D Semantic Tokenizer

2026-05-01

AI Summary

This work addresses the longstanding trade-off between generation quality and reconstruction fidelity in autoregressive image synthesis, which traditionally relies on visual tokenizers trained in a separate, multi-stage pipeline. To overcome this limitation, the authors propose the first end-to-end joint training framework that simultaneously optimizes a 1D semantic tokenizer and an autoregressive generative model. Central to their approach is the integration of a vision foundation model to enrich the semantic expressiveness of the learned tokens, along with a novel mechanism that directly supervises tokenizer learning through the generated outputs—thereby departing from conventional two-stage paradigms. Evaluated on class-unconditional ImageNet generation at 256×256 resolution, the method achieves a new state-of-the-art FID of 1.48.

Abstract

Autoregressive image modeling relies on visual tokenizers to compress images into compact latent representations. We design an end-to-end training pipeline that jointly optimizes reconstruction and generation, enabling direct supervision from generation results to the tokenizer. This contrasts with prior two-stage approaches that train tokenizers and generative models separately. We further investigate leveraging vision foundation models to improve 1D tokenizers for autoregressive modeling. Our autoregressive generative model achieves strong empirical results, including a state-of-the-art FID score of 1.48 without guidance on ImageNet 256x256 generation.

Problem

• autoregressive image generation
• visual tokenizer
• end-to-end training
• 1D semantic tokenizer
• vision foundation models

Innovation

• end-to-end training
• autoregressive image generation
• 1D semantic tokenizer
• vision foundation models
• joint optimization