Quality-Aware Modulation for Diffusion Transformers

2026-06-29

AI Summary

Existing diffusion models often lack explicit awareness of image quality during the denoising process, leading to misaligned outputs, visual inconsistencies, and insufficient fidelity. To address this limitation, this work proposes a Quality Representation Module (QRM), which employs a lightweight Transformer to learn quality-aware representations conditioned on both textual prompts and timesteps. These representations modulate the adaptive LayerNorm layers within a Diffusion Transformer (DiT), thereby injecting quality-sensitive signals into the denoising dynamics. Notably, QRM introduces—for the first time—a lightweight quality-aware modulation mechanism into DiT architectures without altering the sampling strategy or backbone structure. Extensive experiments demonstrate that QRM consistently enhances image quality across multiple DiT baselines, and ablation studies confirm the effectiveness of its loss formulation and architectural design.

Abstract

Modern text-to-image diffusion models, such as diffusion transformers (DiT), rely on timestep or prompt embeddings to modulate the strength of the denoising process in each timestep. While this modulation communicates the current noise level, it does not provide any quality-aware information, which can lead to generated images that are unaligned, visually inconsistent, and lacking in fidelity. In this paper, we propose the Quality Representation Module (QRM), a lightweight transformer module that learns a quality-aware representation based on existing model inputs, and produces a set of vectors $M_{qrm}$. These vectors adjust the adaptive LayerNorm modulation within the DiT transformer blocks, thereby injecting a quality-sensitive signal into the denoising parameters. The QRM introduces no significant changes to the sampling schedule or diffusion backbone. Experiments include ablations on QRM training losses and architectures, as well as empirical results demonstrating consistent image quality improvements over baseline DiT-based models.

Problem

• diffusion transformers
• image quality
• quality-aware modulation
• text-to-image generation
• denoising process

Innovation

• Quality-Aware Modulation
• Diffusion Transformers
• Quality Representation Module
• Adaptive LayerNorm
• Image Generation