Reference-Guided Identity Preserving Face Restoration

2025-05-28

AI Summary

In reference-guided face image restoration using diffusion models, preserving identity consistency with reference faces remains challenging. This paper proposes a training-free, multi-reference adaptive inference framework. Its key contributions are: (1) a composite contextual representation that jointly encodes global semantics and local details from reference faces; (2) a hard-sample identity loss that explicitly enforces discriminative alignment between generated and reference faces in deep feature space; and (3) a reference-guided multi-level feature fusion mechanism. Evaluated on FFHQ-Ref and CelebA-Ref-Test, the method achieves state-of-the-art identity fidelity and reconstruction quality—significantly outperforming existing reference-driven diffusion approaches. It attains superior performance across quantitative metrics (LPIPS, ID-Sim) and user studies, demonstrating both robust identity preservation and high perceptual quality.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Preserving face identity is a critical yet persistent challenge in diffusion-based image restoration. While reference faces offer a path forward, existing reference-based methods often fail to fully exploit their potential. This paper introduces a novel approach that maximizes reference face utility for improved face restoration and identity preservation. Our method makes three key contributions: 1) Composite Context, a comprehensive representation that fuses multi-level (high- and low-level) information from the reference face, offering richer guidance than prior singular representations. 2) Hard Example Identity Loss, a novel loss function that leverages the reference face to address the identity learning inefficiencies found in the existing identity loss. 3) A training-free method to adapt the model to multi-reference inputs during inference. The proposed method demonstrably restores high-quality faces and achieves state-of-the-art identity preserving restoration on benchmarks such as FFHQ-Ref and CelebA-Ref-Test, consistently outperforming previous work.

Problem

• Enhancing identity preservation in diffusion-based face restoration
• Maximizing reference face utility for improved restoration
• Addressing inefficiencies in identity learning with novel losses

Innovation

• Composite Context fuses multi-level reference face information
• Hard Example Identity Loss improves identity learning efficiency
• Training-free adaptation for multi-reference inputs during inference