PIU: Proximity-guided Identity Unlearning in ID-Conditioned Diffusion Models

2026-05-21

AI Summary

This work addresses the challenge of effectively unlearning individual identities in identity-conditioned diffusion models while preserving privacy. The authors propose a forgetting method based on identity embedding replacement and localized fine-tuning. Leveraging the geometric properties of the ArcFace embedding space, they substitute the target identity with a carefully selected neighboring anchor identity, guided by a novel identity replacement objective and a proximity-aware anchor selection mechanism. To minimize disruption to the model’s overall behavior, only the identity-sensitive cross-attention layers are locally fine-tuned. This approach achieves precise, replacement-based identity forgetting for the first time in identity-conditioned diffusion models, significantly suppressing generation of the target identity while maintaining the fidelity and consistency of all other identities. Experimental results demonstrate superior performance over existing methods in both forgetting efficacy and generated image quality.

Abstract

Identity-conditioned diffusion models enable high-quality and identity-consistent face generation, but they also raise severe privacy concerns, as models may continue to synthesize individuals despite their right to be forgotten. While machine unlearning has been extensively studied for concept and data removal, identity unlearning remains largely unexplored, particularly in models conditioned directly on identity embeddings rather than text prompts. In this work, we study identity unlearning in Arc2Face, a state-of-the-art identity-conditioned latent diffusion model for face generation, and introduce Proximity-guided Identity Unlearning (PIU), an anchor-guided framework for identity unlearning. Specifically, we formulate identity removal as an identity replacement objective that reassigns the source identity to a selected anchor identity in the learned identity space, and we complement it with a proximity-based anchor selection strategy motivated by the geometry of ArcFace representations. We further show that effective unlearning can be achieved through localized fine-tuning of a small subset of identity-sensitive cross-attention layers. Experiments across many target identities show that our framework effectively suppresses generation of the target identity while preserving realism and identity consistency for retained identities, as validated by improved performance on unlearning and image-quality metrics, together with qualitative evaluation. The source code for the PIU framework is publicly available at https://github.com/edgarcancinoe/piu_unlearning .

Problem

• identity unlearning
• diffusion models
• privacy
• right to be forgotten
• face generation

Innovation

• identity unlearning
• diffusion models
• proximity-guided
• anchor selection
• cross-attention fine-tuning