DiffusionGuard: A Robust Defense Against Malicious Diffusion-based Image Editing

2024-10-08

International Conference on Learning Representations

AI Summary

Diffusion-based text-guided image editing achieves remarkable performance but is vulnerable to misuse for generating misleading content—particularly in mask-guided editing, where existing adversarial defenses (e.g., post-hoc noise injection) lack robustness. To address this, we propose the first robust defense targeting the early diffusion stages. Our method formulates a mechanism-driven adversarial noise optimization objective and incorporates mask randomization augmentation to enhance generalization. Furthermore, we introduce the first comprehensive evaluation benchmark covering realistic privacy threats. Extensive experiments demonstrate that our approach consistently outperforms the strongest baselines across five critical dimensions: defense success rate, computational efficiency, cross-model transferability, resilience against denoising attacks, and mask generalization. Notably, it achieves up to 28.7% higher defense success under adaptive attacks while reducing inference latency by 39% compared to state-of-the-art alternatives. The benchmark and code are publicly released to foster reproducible research in secure diffusion editing.

Technology Category

• Computer Vision: Adversarial Attacks & Robustness
• Machine Learning: Adversarial Learning & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Security and Privacy: Large-scale security measurements
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Recent advances in diffusion models have introduced a new era of text-guided image manipulation, enabling users to create realistic edited images with simple textual prompts. However, there is significant concern about the potential misuse of these methods, especially in creating misleading or harmful content. Although recent defense strategies, which introduce imperceptible adversarial noise to induce model failure, have shown promise, they remain ineffective against more sophisticated manipulations, such as editing with a mask. In this work, we propose DiffusionGuard, a robust and effective defense method against unauthorized edits by diffusion-based image editing models, even in challenging setups. Through a detailed analysis of these models, we introduce a novel objective that generates adversarial noise targeting the early stage of the diffusion process. This approach significantly improves the efficiency and effectiveness of adversarial noises. We also introduce a mask-augmentation technique to enhance robustness against various masks during test time. Finally, we introduce a comprehensive benchmark designed to evaluate the effectiveness and robustness of methods in protecting against privacy threats in realistic scenarios. Through extensive experiments, we show that our method achieves stronger protection and improved mask robustness with lower computational costs compared to the strongest baseline. Additionally, our method exhibits superior transferability and better resilience to noise removal techniques compared to all baseline methods. Our source code is publicly available at https://github.com/choi403/DiffusionGuard.

Problem

• Defending against malicious diffusion-based image editing
• Protecting images from unauthorized text-guided manipulations
• Enhancing robustness against masked editing attacks

Innovation

• Generates adversarial noise targeting early diffusion stages
• Introduces mask-augmentation technique for enhanced robustness
• Creates comprehensive benchmark for privacy protection evaluation