Variational Latent Entropy Estimation Disentanglement: Controlled Attribute Leakage for Face Recognition

2026-04-13

AI Summary

This work addresses the entanglement of sensitive attributes—such as gender and race—with identity representations in face recognition embeddings, which poses privacy risks and exacerbates fairness concerns. The authors propose VLEED, a post-processing method based on a variational autoencoder that transforms pretrained embeddings by estimating the entropy of sensitive attributes in the latent space and minimizing mutual information between identity and sensitive factors. This approach enables fine-grained control over the degree of sensitive information removal. Experiments on IJB-C, RFW, and VGGFace2 demonstrate that VLEED significantly reduces the predictability of sensitive attributes while preserving high identification accuracy and mitigating demographic disparities, thereby achieving a flexible trade-off between privacy preservation and model utility.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Face recognition embeddings encode identity, but they also encode other factors such as gender and ethnicity. Depending on how these factors are used by a downstream system, separating them from the information needed for verification is important for both privacy and fairness. We propose Variational Latent Entropy Estimation Disentanglement (VLEED), a post-hoc method that transforms pretrained embeddings with a variational autoencoder and encourages a distilled representation where the categorical variable of interest is separated from identity-relevant information. VLEED uses a mutual information-based objective realised through the estimation of the entropy of the categorical attribute in the latent space, and provides stable training with fine-grained control over information removal. We evaluate our method on IJB-C, RFW, and VGGFace2 for gender and ethnicity disentanglement, and compare it to various state-of-the-art methods. We report verification utility, predictability of the disentangled variable under linear and nonlinear classifiers, and group disparity metrics based on false match rates. Our results show that VLEED offers a wide range of privacy-utility tradeoffs over existing methods and can also reduce recognition bias across demographic groups.

Problem

• face recognition
• disentanglement
• privacy
• fairness
• attribute leakage

Innovation

• disentanglement
• variational autoencoder
• mutual information
• privacy-fairness tradeoff
• face recognition bias