Structure Disruption: Subverting Malicious Diffusion-Based Inpainting via Self-Attention Query Perturbation

2025-05-26

AI Summary

Diffusion models for mask-guided image inpainting pose societal risks when maliciously exploited—e.g., to forge sensitive regions. Method: We propose Structural Disruption Attack (SDA), a novel defense mechanism that exploits the sensitivity of diffusion models’ self-attention mechanisms to object contours. SDA injects precise perturbations into query vectors during early denoising steps to disrupt structural reconstruction, integrating mask-aware, step-targeted perturbation scheduling and local robustness regularization. Contribution/Results: SDA achieves structure-level protection while preserving visual naturalness and high fidelity. It significantly reduces malicious inpainting success rates, attains state-of-the-art defensive performance across multiple benchmark datasets (e.g., Places2, CelebA-HQ), and demonstrates strong cross-model generalization robustness.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

The rapid advancement of diffusion models has enhanced their image inpainting and editing capabilities but also introduced significant societal risks. Adversaries can exploit user images from social media to generate misleading or harmful content. While adversarial perturbations can disrupt inpainting, global perturbation-based methods fail in mask-guided editing tasks due to spatial constraints. To address these challenges, we propose Structure Disruption Attack (SDA), a powerful protection framework for safeguarding sensitive image regions against inpainting-based editing. Building upon the contour-focused nature of self-attention mechanisms of diffusion models, SDA optimizes perturbations by disrupting queries in self-attention during the initial denoising step to destroy the contour generation process. This targeted interference directly disrupts the structural generation capability of diffusion models, effectively preventing them from producing coherent images. We validate our motivation through visualization techniques and extensive experiments on public datasets, demonstrating that SDA achieves state-of-the-art (SOTA) protection performance while maintaining strong robustness.

Problem

• Prevent malicious image inpainting via self-attention query perturbation
• Disrupt structural generation in diffusion models to block harmful edits
• Protect sensitive image regions from misleading AI-based manipulations

Innovation

• Optimizes perturbations disrupting self-attention queries
• Targets initial denoising step to destroy contours
• Achieves SOTA protection with strong robustness