Self-Rectifying Diffusion Sampling with Perturbed-Attention Guidance

2024-03-26

European Conference on Computer Vision

AI Summary

Diffusion models face limitations in unconditional generation and image restoration due to conventional sampling guidance—e.g., classifier guidance—requiring auxiliary training or external modules, thus lacking a universal, lightweight guidance mechanism. To address this, we propose Perturbed Attention Guidance (PAG), the first method leveraging the structural representational capacity of self-attention within diffusion U-Nets: by replacing attention maps with identity matrices to construct structurally degenerated samples, PAG inversely steers the denoising process via self-calibrating sampling—requiring no additional parameters or fine-tuning. The approach involves only attention perturbation, identity matrix injection, and progressive structural guidance. Evaluated on ADM and Stable Diffusion, PAG significantly improves both conditional and unconditional generation quality. Moreover, it substantially outperforms Classifier Guidance (CG) and Classifier-Free Guidance (CFG) baselines across diverse tasks—including ControlNet with null prompts, image inpainting, and deblurring—demonstrating broad applicability and efficacy.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Recent studies have demonstrated that diffusion models are capable of generating high-quality samples, but their quality heavily depends on sampling guidance techniques, such as classifier guidance (CG) and classifier-free guidance (CFG). These techniques are often not applicable in unconditional generation or in various downstream tasks such as image restoration. In this paper, we propose a novel sampling guidance, called Perturbed-Attention Guidance (PAG), which improves diffusion sample quality across both unconditional and conditional settings, achieving this without requiring additional training or the integration of external modules. PAG is designed to progressively enhance the structure of samples throughout the denoising process. It involves generating intermediate samples with degraded structure by substituting selected self-attention maps in diffusion U-Net with an identity matrix, by considering the self-attention mechanisms' ability to capture structural information, and guiding the denoising process away from these degraded samples. In both ADM and Stable Diffusion, PAG surprisingly improves sample quality in conditional and even unconditional scenarios. Moreover, PAG significantly improves the baseline performance in various downstream tasks where existing guidances such as CG or CFG cannot be fully utilized, including ControlNet with empty prompts and image restoration such as inpainting and deblurring.

Problem

• Improves diffusion sample quality without extra training
• Enhances structure in unconditional and conditional generation
• Boosts performance in downstream tasks like image restoration

Innovation

• Perturbed-Attention Guidance (PAG) enhances diffusion sampling
• PAG substitutes self-attention maps with identity matrix
• PAG improves quality in unconditional and conditional tasks