An Empirical Study of Training Pixel-Space Text-to-Image Diffusion Models

2026-08-17

AI Summary

This study addresses the low training efficiency of pixel-space diffusion models by proposing a latent-to-pixel space transfer strategy. Through systematic optimization of weight initialization, data composition, noise scheduling, and decoder architecture, we establish an efficient pixel-space training paradigm. Experiments demonstrate that this approach effectively resolves slow pre-training convergence, achieving performance that matches or surpasses latent-space models while improving end-to-end inference speed by 3.18× to 4.75×. Consequently, this work provides a comprehensive guideline for high-performance and practical pixel-level training in high-resolution image generation, bridging the gap between theoretical efficacy and computational efficiency in generative modeling.

Abstract

This paper investigates an increasingly important topic in generative modeling: pixel-space diffusion models. Although numerous studies have explored this topic, most focus on small-scale or class-conditional settings. Consequently, a practical recipe for training pixel-space models that rival or exceed well-established latent-space counterparts remains elusive. Through a comprehensive empirical study, we first observe that direct large-scale pre-training in pixel space converges substantially more slowly than in latent space. This observation motivates a latent-to-pixel strategy that acquires generative priors efficiently in latent space and transitions to pixel space during post-training. We then systematically investigate the key design choices governing this transition, including weight initialization, data composition, prediction target, decoder architecture, and noise schedule, and identify a practical recipe that makes the resulting pixel-space models match or outperform their latent-space counterparts while delivering 3.18 to 4.75 times end-to-end inference speedups. We hope that our findings provide useful empirical insights and practical guidelines for future research on pixel-space generation.

Problem

• Pixel-space diffusion models
• Text-to-image generation
• Large-scale training
• Training recipe

Innovation

• Pixel-Space Diffusion
• Latent-to-Pixel Strategy
• Training Recipe
• Inference Speedup
• Empirical Study