VariFace: Fair and Diverse Synthetic Dataset Generation for Face Recognition

2024-12-09

arXiv.org

AI Summary

Existing face recognition models rely heavily on large-scale web-scraped datasets, raising serious privacy concerns and exacerbating algorithmic bias; meanwhile, current synthetic face data suffer from insufficient intra-class and inter-class diversity, resulting in substantially lower performance than models trained on real data. This paper proposes the first framework for constructing synthetic face datasets that explicitly prioritizes fairness and diversity. Our method employs a two-stage diffusion-based generation architecture integrating: (i) consistency-aware label refinement for face identity preservation, (ii) Face Vendi Score-guided optimization of inter-class diversity, and (iii) divergence-score-constrained conditional control in latent space. Evaluated at comparable scale, our synthetic dataset outperforms all prior synthetic benchmarks (0.9200 → 0.9405), and—critically—is the first to surpass the widely used real-world CASIA-WebFace dataset across all six standard benchmarks, achieving a mean verification accuracy of 0.9567.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Machine Learning: Ethics, Bias, and Fairness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

The use of large-scale, web-scraped datasets to train face recognition models has raised significant privacy and bias concerns. Synthetic methods mitigate these concerns and provide scalable and controllable face generation to enable fair and accurate face recognition. However, existing synthetic datasets display limited intraclass and interclass diversity and do not match the face recognition performance obtained using real datasets. Here, we propose VariFace, a two-stage diffusion-based pipeline to create fair and diverse synthetic face datasets to train face recognition models. Specifically, we introduce three methods: Face Recognition Consistency to refine demographic labels, Face Vendi Score Guidance to improve interclass diversity, and Divergence Score Conditioning to balance the identity preservation-intraclass diversity trade-off. When constrained to the same dataset size, VariFace considerably outperforms previous synthetic datasets (0.9200 $ ightarrow$ 0.9405) and achieves comparable performance to face recognition models trained with real data (Real Gap = -0.0065). In an unconstrained setting, VariFace not only consistently achieves better performance compared to previous synthetic methods across dataset sizes but also, for the first time, outperforms the real dataset (CASIA-WebFace) across six evaluation datasets. This sets a new state-of-the-art performance with an average face verification accuracy of 0.9567 (Real Gap = +0.0097) across LFW, CFP-FP, CPLFW, AgeDB, and CALFW datasets and 0.9366 (Real Gap = +0.0380) on the RFW dataset.

Problem

• Address privacy and bias in face recognition datasets
• Enhance intraclass and interclass diversity in synthetic faces
• Improve face recognition performance using synthetic data

Innovation

• Two-stage diffusion-based synthetic face generation
• Face Recognition Consistency for demographic labels
• Face Vendi Score for interclass diversity