IREU: Identity-Related Encoder-Only Unlearning for Customized Portrait Generation

2026-06-29

AI Summary

This work addresses the critical issue of identity privacy leakage in personalized portrait generation caused by model misuse. To mitigate this risk, the authors propose a precise identity unlearning method that operates without fine-tuning: it first identifies, in an offline manner, the feature regions within the image encoder that are associated with a target identity and then applies selective perturbations to suppress only the generation capability of that specific identity. Crucially, this approach preserves high-fidelity generation for all other identities and generalizes directly across different generators sharing the same encoder. By enabling localized, non-intrusive identity removal, the method achieves effective prevention of identity misuse while maintaining overall generation fidelity.

Abstract

Customized Portrait Generation (CPG) technologies have been widely used to generate high-fidelity person images given an input image indicating the identity and a text prompt indicating the required edits. Yet these methods pose significant privacy risks by spreading fake visual information. Against such risks, each public generator should be able to suppress its generation ability for a particular person when requested. Therefore, in this work we investigate the identity unlearning problem for CPG. Since there are no previous methods in this field, we propose a simple baseline that updates the image encoder by minimizing identity similarity between generated and input images for target identities to be unlearned, while maximizing it for identities to be retained. However, we find such a global perturbation in the feature space harms the fidelity of generated images for other identities to be retained. To solve this problem, we propose a novel method IREU, which first locates identity-related features in an offline manner and then only performs feature perturbations on them. The experimental results show that our proposed method IREU achieves better identity unlearning performance for target identities to be unlearned, and also keeps high fidelity for other identities to be retained. In addition, our unlearned image encoder is generalizable across different generators with the same encoder without fine-tuning, which is friendly for deployment in practice.

Problem

• identity unlearning
• customized portrait generation
• privacy risk
• image generation
• identity suppression

Innovation

• identity unlearning
• customized portrait generation
• feature perturbation
• encoder-only unlearning
• privacy-preserving generation