Diff-PC: Identity-preserving and 3D-aware controllable diffusion for zero-shot portrait customization

2024-12-01

Information Fusion

AI Summary

Existing portrait customization methods often struggle to simultaneously preserve identity fidelity and achieve precise facial control. To address this limitation, this work proposes Diff-PC, a novel framework that leverages a 3D face-guided identity encoder and a feature injection mechanism to enable high-fidelity, fine-grained controllable portrait generation under zero-shot conditions. The approach effectively integrates 3D-aware priors with identity features and is compatible with diverse backgrounds and multi-style base models. Trained with a dedicated identity-centric dataset and enhanced by an ID-Encoder, an ID-Ctrl alignment module, and an ID-Injector refinement module, Diff-PC consistently outperforms state-of-the-art methods in terms of identity preservation, facial controllability, and text-to-image alignment.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data

Problem

• portrait customization
• identity preservation
• facial control
• zero-shot generation
• 3D-aware

Innovation

• 3D-aware diffusion
• identity preservation
• zero-shot portrait customization
• facial controllability
• ID-Encoder