DiT-Air: Revisiting the Efficiency of Diffusion Model Architecture Design in Text to Image Generation

2025-03-13

AI Summary

Diffusion Transformers (DiTs) face efficiency bottlenecks in text-to-image generation, particularly concerning parameter efficiency and computational overhead. Method: We systematically evaluate architectural design, text-conditioning mechanisms, and training strategies, revealing that standard DiT outperforms complex variants in both performance and parameter efficiency. Based on these insights, we propose DiT-Air—a lightweight, high-efficiency architecture—and its ultra-light variant, DiT-Air-Lite—featuring inter-layer parameter sharing, reducing model size by 66% versus MMDiT. We further optimize joint training of the text encoder and VAE, and integrate supervised fine-tuning with reward-driven fine-tuning to enhance generation quality. Contribution/Results: DiT-Air achieves state-of-the-art performance on GenEval and T2I CompBench. DiT-Air-Lite attains superior results over most existing models despite its minimal parameter count, demonstrating the effectiveness and feasibility of streamlined DiT design.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In this work, we empirically study Diffusion Transformers (DiTs) for text-to-image generation, focusing on architectural choices, text-conditioning strategies, and training protocols. We evaluate a range of DiT-based architectures--including PixArt-style and MMDiT variants--and compare them with a standard DiT variant which directly processes concatenated text and noise inputs. Surprisingly, our findings reveal that the performance of standard DiT is comparable with those specialized models, while demonstrating superior parameter-efficiency, especially when scaled up. Leveraging the layer-wise parameter sharing strategy, we achieve a further reduction of 66% in model size compared to an MMDiT architecture, with minimal performance impact. Building on an in-depth analysis of critical components such as text encoders and Variational Auto-Encoders (VAEs), we introduce DiT-Air and DiT-Air-Lite. With supervised and reward fine-tuning, DiT-Air achieves state-of-the-art performance on GenEval and T2I CompBench, while DiT-Air-Lite remains highly competitive, surpassing most existing models despite its compact size.

Problem

• Evaluates Diffusion Transformers for text-to-image generation efficiency.
• Compares standard DiT with specialized models for parameter efficiency.
• Introduces DiT-Air and DiT-Air-Lite for state-of-the-art performance.

Innovation

• Layer-wise parameter sharing reduces model size.
• DiT-Air achieves state-of-the-art performance with fine-tuning.
• Standard DiT shows superior parameter-efficiency when scaled.