An h-space Based Adversarial Attack for Protection Against Few-shot Personalization

2025-07-23

AI Summary

Diffusion models in few-shot personalization pose significant privacy risks by inadvertently leaking sensitive user image content. To address this, we propose HAAD—the first adversarial defense method targeting the semantic latent space (h-space) of diffusion models—which injects carefully crafted perturbations into h-space to precisely disrupt the model’s ability to generate personalized concepts. We further introduce HAAD-KV, a lightweight variant that optimizes only the key-value parameters in attention layers, substantially reducing computational overhead while enhancing robustness against adversarial attacks. Extensive experiments across diverse few-shot personalization scenarios demonstrate that HAAD effectively suppresses unauthorized content generation, achieving an average 12.7% improvement in defense success rate over state-of-the-art methods and reducing inference latency by 43%. Our approach establishes an efficient, deployable paradigm for privacy-preserving diffusion modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents

Application Category

• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search

Abstract

The versatility of diffusion models in generating customized images from few samples raises significant privacy concerns, particularly regarding unauthorized modifications of private content. This concerning issue has renewed the efforts in developing protection mechanisms based on adversarial attacks, which generate effective perturbations to poison diffusion models. Our work is motivated by the observation that these models exhibit a high degree of abstraction within their semantic latent space (`h-space'), which encodes critical high-level features for generating coherent and meaningful content. In this paper, we propose a novel anti-customization approach, called HAAD (h-space based Adversarial Attack for Diffusion models), that leverages adversarial attacks to craft perturbations based on the h-space that can efficiently degrade the image generation process. Building upon HAAD, we further introduce a more efficient variant, HAAD-KV, that constructs perturbations solely based on the KV parameters of the h-space. This strategy offers a stronger protection, that is computationally less expensive. Despite their simplicity, our methods outperform state-of-the-art adversarial attacks, highlighting their effectiveness.

Problem

• Prevent unauthorized image customization using diffusion models
• Protect privacy by poisoning diffusion models with perturbations
• Leverage h-space features to degrade image generation quality

Innovation

• Leverages h-space for adversarial attack
• Uses KV parameters for efficient perturbation
• Outperforms state-of-the-art adversarial attacks