Unified Backbone Refinement for Diffusion Models via Internal-Latent Analysis

2026-07-04

AI Summary

This work addresses a critical yet previously unexamined issue in diffusion models: abrupt changes in deep latent variables during early generation stages often lead to image artifacts and hallucinations. The study is the first to establish a clear link between such latent-space discontinuities and generation defects. To mitigate this, the authors propose DUNE, a training-free framework that leverages phase-aware analysis and an Exponential Moving Average (EMA)-based criterion to identify anomalous latent variables, followed by an adaptive suppression strategy tailored to the backbone architecture. DUNE is universally applicable across both U-Net and Transformer-based diffusion models, consistently enhancing generation fidelity and substantially alleviating hallucination across diverse model backbones.

Abstract

Diffusion models have achieved remarkable success across diverse domains, with performance closely related to the denoising backbones that parameterize the score function. In this paper, we present a systematic, phase-aware analysis of diffusion components and show that abrupt, early-stage fluctuations in deep latents are strongly associated with artifacts. Guided by these findings, we introduce DUNE (Diffusion Unified Network refiNEr), a training-free refinement framework that detects abrupt deviations in deep low-noise internal latents using a shared EMA-based criterion, and applies backbone-specific suppression to the detector-selected entries. Although derived from U-Net, the same detect-suppress principle extends naturally to Transformer-based diffusion models by acting on the latents of deep self-attention blocks. Extensive experiments across multiple backbones indicate that DUNE improves fidelity while reducing hallucinations, offering new insight into where and when diffusion backbones should be controlled.

Problem

• diffusion models
• denoising backbone
• latent fluctuations
• artifacts
• hallucinations

Innovation

• diffusion models
• latent analysis
• training-free refinement
• artifact suppression
• backbone-agnostic