Unveiling the Secret of AdaLN-Zero in Diffusion Transformer

2026-08-10

AI Summary

This work investigates the fundamental reason behind the superior performance of the adaLN-Zero mechanism in diffusion Transformers and finds that its efficacy primarily stems from the zero-initialization strategy, rather than the SE-like architecture or the “progressive” update order. Building on this insight, the authors propose adaLN-Gaussian initialization and an enhanced SE-adaLN-Zero mechanism, which jointly improve training efficiency and generation quality while preserving theoretical interpretability. Experimental results across four benchmarks, including ImageNet1K, demonstrate that the proposed approaches significantly accelerate convergence and enhance image generation fidelity. Furthermore, the methods exhibit strong generalization capabilities in text-to-image synthesis tasks.

Abstract

Diffusion transformer (DiT), a rapidly emerging architecture for image generation, has gained much attention. However, despite ongoing efforts to improve its performance, the understanding of DiT remains superficial. In this work, we delve into and investigate a critical conditioning mechanism within DiT, adaLN-Zero, which achieves superior performance compared to adaLN. Our work studies three potential elements driving this performance, including an SE-like structure, zero-initialization, and a "gradual" update order, among which zero-initialization is proved to be the most influential. Building on this understanding, we propose an analysis-guided initialization strategy, termed adaLN-Gaussian, which serves both as an empirical validation of our analysis and as a practical initialization method that consistently improves optimization efficiency. On the other hand, inspired by the SE-like structure, we introduce an improved conditioning mechanism called SE-adaLN-Zero. Extensive experiments following DiT on four datasets, especially on ImageNet1K demonstrate the effectiveness and generalization of adaLN-Gaussian and SE-adaLN-Zero. Beyond class-to-image generation, we also evaluate the generalization of the two improved methods on text-to-image generation.

Problem

• Diffusion Transformer
• adaLN-Zero
• conditioning mechanism
• zero-initialization
• image generation

Innovation

• adaLN-Zero
• zero-initialization
• Diffusion Transformer
• conditioning mechanism
• adaLN-Gaussian