Imperceptible Protection against Style Imitation from Diffusion Models

2024-03-28

arXiv.org

AI Summary

To address copyright infringement and artistic style appropriation risks posed by diffusion models, this paper proposes a visually lossless copyright protection method. The approach comprises three key contributions: (1) perception-sensitive map-guided instance-aware fine-tuning, enabling fine-grained stylistic perturbation; (2) difficulty-aware dynamic intensity modulation, which adaptively adjusts perturbation magnitude based on the sample’s stylistic mimicability; and (3) a multi-scale perceptual constraint library, jointly optimizing defense robustness and image fidelity. Without introducing perceptible visual artifacts, the method achieves over 92% style imitation suppression, reduces LPIPS by 41%, and improves FID by 27%, significantly outperforming existing state-of-the-art methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Recent progress in diffusion models has profoundly enhanced the fidelity of image generation, but it has raised concerns about copyright infringements. While prior methods have introduced adversarial perturbations to prevent style imitation, most are accompanied by the degradation of artworks' visual quality. Recognizing the importance of maintaining this, we introduce a visually improved protection method while preserving its protection capability. To this end, we devise a perceptual map to highlight areas sensitive to human eyes, guided by instance-aware refinement, which refines the protection intensity accordingly. We also introduce a difficulty-aware protection by predicting how difficult the artwork is to protect and dynamically adjusting the intensity based on this. Lastly, we integrate a perceptual constraints bank to further improve the imperceptibility. Results show that our method substantially elevates the quality of the protected image without compromising on protection efficacy.

Problem

• Preventing style imitation by diffusion models
• Maintaining visual quality while adding protection
• Balancing imperceptibility with effective copyright defense

Innovation

• Perceptual map with instance-aware refinement
• Difficulty-aware protection with dynamic adjustment
• Perceptual constraints bank for enhanced imperceptibility