Generating Synthetic Data via Augmentations for Improved Facial Resemblance in DreamBooth and InstantID

2025-05-06

AI Summary

This work addresses the insufficient facial identity preservation in professional portrait generation using DreamBooth and InstantID. We propose a face-centric data augmentation strategy for personalized fine-tuning. Methodologically, we introduce FaceDistance—a novel evaluation framework based on FaceNet embeddings—to systematically quantify the impact of geometric, illumination, and stylization augmentations on SDXL-generated portraits, identifying the optimal augmentation combination. Our contributions are twofold: (1) establishing a reproducible, quantitative benchmark for facial identity fidelity; and (2) revealing how targeted augmentations enhance robustness across pose and lighting variations. Experiments demonstrate an average 18.7% improvement in FaceDistance scores, significantly strengthening facial consistency between generated outputs and source images. The approach provides an effective technical pathway for generating high-fidelity professional portraits from low-quality inputs.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The personalization of Stable Diffusion for generating professional portraits from amateur photographs is a burgeoning area, with applications in various downstream contexts. This paper investigates the impact of augmentations on improving facial resemblance when using two prominent personalization techniques: DreamBooth and InstantID. Through a series of experiments with diverse subject datasets, we assessed the effectiveness of various augmentation strategies on the generated headshots' fidelity to the original subject. We introduce FaceDistance, a wrapper around FaceNet, to rank the generations based on facial similarity, which aided in our assessment. Ultimately, this research provides insights into the role of augmentations in enhancing facial resemblance in SDXL-generated portraits, informing strategies for their effective deployment in downstream applications.

Problem

• Improving facial resemblance in Stable Diffusion-generated portraits
• Assessing augmentation impact on DreamBooth and InstantID techniques
• Developing FaceDistance metric for facial similarity ranking

Innovation

• Uses augmentations to improve facial resemblance
• Introduces FaceDistance for facial similarity ranking
• Assesses augmentation strategies in DreamBooth and InstantID