SteerFace: Debiasing Synthetic Face Generation via Adaptive Residue Perturbation

2026-05-29

AI Summary

This work addresses the performance degradation of face recognition models in synthetic-to-real scenarios, primarily caused by visual bias in synthetic face data that deviates from the true data distribution. To mitigate this issue, the authors propose SteerFace, a framework that introduces an identity-preserving regularizer during diffusion-based generation by applying adaptive hyperspherical orthogonal perturbations to identity embeddings. This approach explicitly curbs the generator’s reliance on non-identity visual cues. SteerFace is the first method to systematically identify and tackle visual bias in synthetic faces, demonstrating consistently superior face recognition performance and strong generalization across multiple datasets and generative pipelines, thereby significantly narrowing the performance gap between synthetic and real face data.

Abstract

The shortage of legally compliant data for face recognition training has sparked growing interest in using synthetic data as an alternative. While recent diffusion-based methods enable the generation of photorealistic face images with strong identity adherence and data diversity, their downstream recognition performance still exhibits a significant synthetic-real gap. This paper identifies visual tendency as a previously underexplored limitation, whereby synthetic data exhibit an unrealistic prevalence of visual attributes and thus deviate from the real-data distribution. Visual tendency can be attributed to the generator's conditioning on identity embeddings, through which co-occurring residual visual cues are unintentionally absorbed into learned identity semantics. To discourage the generator from exploiting such visual cues, this paper proposes SteerFace, a simple and efficient training framework that perturbs identity embeddings by steering them toward random orthogonal directions on the embedding hypersphere. The perturbation serves as an identity-preserving regularizer that penalizes the generator's reliance on non-identity components, as supported by theoretical analysis. This paper further introduces an adaptive strategy that learns perturbation strengths with both sample-wise preference and favorable overall statistics. Extensive experiments show that SteerFace effectively mitigates visual tendency, outperforms prior methods in downstream face recognition, and generalizes well across different training datasets and generation pipelines.

Problem

• synthetic face generation
• visual tendency
• face recognition
• data bias
• identity embedding

Innovation

• adaptive residue perturbation
• synthetic face generation
• visual tendency
• identity embedding regularization
• diffusion-based generation