Parallel Rollout Approximation for Pixel-Space Autoregressive Image Generation

2026-06-26

AI Summary

This work addresses the limitations of pixel-space autoregressive image generation, which suffers from substantial modeling errors due to high-dimensional patch prediction and error accumulation caused by train-inference inconsistency. To overcome these challenges, the authors propose the Parallel Rollback Approximation (PRA) framework, which models generation in a low-dimensional latent state and maps back to pixel space, thereby preserving pixel-level input–output interfaces while emulating inference-time feedback mechanisms and enabling parallelized training. PRA is the first approach to jointly mitigate high-dimensional generation errors and the train-inference gap, enabling scalable pixel-level autoregressive synthesis. On ImageNet-1K 256×256 conditional generation, PRA-S (135M) and PRA-L (511M) achieve FID scores of 2.58 and 1.94, respectively—setting new state-of-the-art results for pixel-space autoregressive models—and outperform both diffusion and autoregressive baselines in classification probe tasks.

Abstract

Pixel-space continuous-token autoregressive (AR) generation directly models images as sequences of raw pixel patches, avoiding discrete tokenization or a separately pretrained tokenizer. However, it faces coupled challenges: high-dimensional patch generation causes large single-step errors, and teacher-forced training creates a train--inference gap that makes these errors accumulate across AR steps. Existing fixes such as $x$-prediction and input noise injection only partially mitigate these issues. Exact rollout training better matches inference-time conditions, but is impractical due to prohibitively slow sequential sampling. We propose \emph{Parallel Rollout Approximation} (PRA), a scalable framework that addresses both challenges jointly. PRA generates low-dimensional intermediate states instead of high-dimensional pixel patches, then maps them back to pixel-space tokens with a pixel decoder, preserving a pixel-in, pixel-out AR interface. It also constructs inference-like pixel inputs through the same intermediate-state-to-pixel path used at inference, independently across positions, approximating the pixel-feedback interface encountered during inference-time rollout while retaining parallel teacher-forced training. On class-conditional ImageNet-1K generation at $256\times256$ resolution, PRA-S with 135M parameters achieves an FID of 2.58, surpassing the previous billion-scale pixel-space AR result of 3.60. Scaling to PRA-L with 511M parameters further improves FID to 1.94, establishing a new state of the art among pixel-space AR models. Beyond generation, PRA achieves higher ImageNet classification probing accuracy than other AR and diffusion baselines, suggesting its potential for unified pixel-space image generation and understanding.

Problem

• autoregressive image generation
• pixel-space modeling
• train-inference gap
• high-dimensional patch generation
• error accumulation

Innovation

• Parallel Rollout Approximation
• pixel-space autoregressive generation
• train-inference gap
• intermediate-state modeling
• scalable image generation