NFIG: Autoregressive Image Generation with Next-Frequency Prediction

2025-03-10

AI Summary

Autoregressive image generation faces challenges including difficulty in modeling long-range dependencies, high computational overhead, and spatial token ordering lacking semantic meaning. This paper proposes NFIG, the first frequency-guided autoregressive framework, which decomposes image synthesis into a multi-stage, spectrally hierarchical process: modeling low-frequency global structure first, then progressively injecting high-frequency details—aligning with the intrinsic spectral decay of natural images. NFIG comprises frequency-domain decomposition, band-wise tokenization and prediction, and progressive reconstruction, eliminating conventional raster-scan token sequences. On ImageNet-256, NFIG achieves a state-of-the-art FID of 2.81, outperforming VAR-d20 by 1.25× speedup in sampling latency while substantially reducing the number of sampling steps. The framework thus delivers both superior modeling fidelity and inference efficiency.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Autoregressive models have achieved promising results in natural language processing. However, for image generation tasks, they encounter substantial challenges in effectively capturing long-range dependencies, managing computational costs, and most crucially, defining meaningful autoregressive sequences that reflect natural image hierarchies. To address these issues, we present extbf{N}ext- extbf{F}requency extbf{I}mage extbf{G}eneration ( extbf{NFIG}), a novel framework that decomposes the image generation process into multiple frequency-guided stages. Our approach first generates low-frequency components to establish global structure with fewer tokens, then progressively adds higher-frequency details, following the natural spectral hierarchy of images. This principled autoregressive sequence not only improves the quality of generated images by better capturing true causal relationships between image components, but also significantly reduces computational overhead during inference. Extensive experiments demonstrate that NFIG achieves state-of-the-art performance with fewer steps, offering a more efficient solution for image generation, with 1.25$ imes$ speedup compared to VAR-d20 while achieving better performance (FID: 2.81) on the ImageNet-256 benchmark. We hope that our insight of incorporating frequency-domain knowledge to guide autoregressive sequence design will shed light on future research. We will make our code publicly available upon acceptance of the paper.

Problem

• Addresses challenges in autoregressive image generation models.
• Improves image quality by capturing true causal relationships.
• Reduces computational overhead with frequency-guided stages.

Innovation

• Decomposes image generation into frequency-guided stages
• Generates low-frequency components first for global structure
• Progressively adds higher-frequency details for image refinement