Refining Visual Artifacts in Diffusion Models via Explainable AI-based Flaw Activation Maps

2025-12-09

AI Summary

Diffusion models often produce generated images containing visual artifacts and implausible regions, limiting their practical deployment. To address this, we propose the first explainable AI (XAI)-driven self-optimization framework for diffusion models. Our method localizes problematic regions in the generation process via defect activation maps (FAMs) and introduces a two-stage noise-augmentation–reverse-repair mechanism: during the forward process, noise is dynamically amplified in defective regions; during the reverse process, denoising is explicitly focused on repairing those regions. Crucially, our approach requires no architectural modification or model retraining, ensuring compatibility across diverse diffusion models and tasks—including unconditional image generation, text-to-image synthesis, and image inpainting. Extensive experiments demonstrate consistent and substantial improvements, with Fréchet Inception Distance (FID) reduced by up to 27.3%. This work establishes the first XAI-guided, proactive optimization of diffusion processes.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Explainable AI (XAI) for Human Understanding
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Diffusion models have achieved remarkable success in image synthesis. However, addressing artifacts and unrealistic regions remains a critical challenge. We propose self-refining diffusion, a novel framework that enhances image generation quality by detecting these flaws. The framework employs an explainable artificial intelligence (XAI)-based flaw highlighter to produce flaw activation maps (FAMs) that identify artifacts and unrealistic regions. These FAMs improve reconstruction quality by amplifying noise in flawed regions during the forward process and by focusing on these regions during the reverse process. The proposed approach achieves up to a 27.3% improvement in Fréchet inception distance across various diffusion-based models, demonstrating consistently strong performance on diverse datasets. It also shows robust effectiveness across different tasks, including image generation, text-to-image generation, and inpainting. These results demonstrate that explainable AI techniques can extend beyond interpretability to actively contribute to image refinement. The proposed framework offers a versatile and effective approach applicable to various diffusion models and tasks, significantly advancing the field of image synthesis.

Problem

• Detects and corrects artifacts in diffusion models
• Uses explainable AI to identify unrealistic image regions
• Improves image generation across various tasks and datasets

Innovation

• Explainable AI flaw activation maps detect artifacts
• Amplify noise in flawed regions during forward process
• Focus on flawed regions during reverse denoising