DCTdiff: Intriguing Properties of Image Generative Modeling in the DCT Space

2024-12-19

arXiv.org

AI Summary

To address the high computational cost and resolution limitations of image generation models operating in the pixel or latent space, this paper proposes DCTdiff—the first end-to-end diffusion framework that directly models and denoises in the discrete cosine transform (DCT) frequency domain. Theoretically, we establish that the diffusion process is equivalent to autoregressive modeling in the spectral domain. Methodologically, DCTdiff eliminates the need for VAE-based latent spaces, adapts UViT/DiT architectures, and integrates multiple samplers to enable full-stack generation entirely within the frequency domain. Experiments demonstrate that DCTdiff achieves superior FID scores over Latent Diffusion (SD-VAE) at 512×512 resolution while reducing training cost to only one-quarter. Furthermore, our analysis systematically uncovers critical design principles and intrinsic properties of spectral-domain modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

This paper explores image modeling from the frequency space and introduces DCTdiff, an end-to-end diffusion generative paradigm that efficiently models images in the discrete cosine transform (DCT) space. We investigate the design space of DCTdiff and reveal the key design factors. Experiments on different frameworks (UViT, DiT), generation tasks, and various diffusion samplers demonstrate that DCTdiff outperforms pixel-based diffusion models regarding generative quality and training efficiency. Remarkably, DCTdiff can seamlessly scale up to 512$ imes$512 resolution without using the latent diffusion paradigm and beats latent diffusion (using SD-VAE) with only 1/4 training cost. Finally, we illustrate several intriguing properties of DCT image modeling. For example, we provide a theoretical proof of why 'image diffusion can be seen as spectral autoregression', bridging the gap between diffusion and autoregressive models. The effectiveness of DCTdiff and the introduced properties suggest a promising direction for image modeling in the frequency space. The code is https://github.com/forever208/DCTdiff.

Problem

• Explores image modeling in frequency space using DCT
• Proposes DCTdiff for better quality and efficiency
• Bridges diffusion and autoregressive models theoretically

Innovation

• DCTdiff models images in frequency space
• Outperforms pixel-based diffusion models
• Scales to 512x512 resolution efficiently