GEAR: Guided End-to-End AutoRegression for Image Synthesis

2026-06-30

AI Summary

This work addresses the performance limitations of conventional two-stage training in visual generative models, where the tokenizer and generator develop misaligned modeling preferences. To overcome this, the authors propose GEAR, a novel approach featuring a hard-soft dual-branch mechanism that enables end-to-end joint training of vector-quantized tokenizers and autoregressive generators through representation alignment. While preserving the autoregressive property, GEAR steers the tokenizer toward learning index distributions that are easier for the generator to predict, effectively shifting the burden of semantic alignment to the generator. The method is compatible with various quantizers—including VQVAE, LFQ, and IBQ—and achieves up to a 10× faster convergence in gFID on ImageNet. It also substantially improves patch-level and spatial consistency and successfully generalizes to text-to-image generation tasks.

Abstract

Visual generative models are typically trained in two stages. A tokenizer is first trained for reconstruction and then frozen, after which a generator is trained on its discrete indices or continuous latents. This decoupling leaves the tokenizer unaware of what the generator finds easy to model. We present GEAR (Guided End-to-end AutoRegression), which trains a vector-quantized (VQ) tokenizer and an autoregressive (AR) generator jointly and end-to-end, guided by representation alignment. The key obstacle is that the VQ index fed to the AR model is non-differentiable, so gradients cannot reach the tokenizer, and a straight-through estimator collapses. GEAR resolves this with a dual read-out of the codebook assignment. A hard, one-hot branch trains the AR with next-token prediction, while a differentiable soft branch carries a representation-alignment loss that flows back to guide only the tokenizer. The AR model thereby steers its tokenizer toward an index distribution it can predict more easily. This shifts the alignment burden from the tokenizer to the AR: the tokenizer's own features become less DINOv2-like while the AR's become more so, the opposite of diffusion-side recipes that make the latent itself semantic. GEAR speeds up ImageNet gFID convergence by up to 10x relative to the strong LlamaGen-REPA baseline, learns markedly better patch-level and spatially-coherent features, and generalizes across quantizers (VQVAE, LFQ, IBQ) and to text-to-image generation.

Problem

• image synthesis
• tokenizer-generator decoupling
• end-to-end training
• vector quantization
• autoregressive generation

Innovation

• end-to-end training
• vector-quantized tokenizer
• autoregressive generation
• representation alignment
• differentiable soft assignment