SynthLight: Portrait Relighting with Diffusion Model by Learning to Re-render Synthetic Faces

2025-01-16

AI Summary

This work addresses physically consistent illumination relighting of portrait images, aiming to naturally adjust highlights, shadows, and global lighting while preserving identity recognizability. We propose a synthesis-driven diffusion relighting paradigm: a PBR-rendered 3D face model generates paired illumination data, enabling a synthetic-to-real domain transfer learning framework; we design a label-free, multi-task supervision strategy on real portraits and incorporate classifier-free guidance sampling to jointly optimize illumination realism and identity fidelity. Our method integrates diffusion modeling, physics-based rendering, domain adaptation, and multi-task learning. On the Light Stage dataset, it achieves state-of-the-art performance. For in-the-wild images, it produces highly realistic relit results with significantly improved identity preservation over existing methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Multimodal Learning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

We introduce SynthLight, a diffusion model for portrait relighting. Our approach frames image relighting as a re-rendering problem, where pixels are transformed in response to changes in environmental lighting conditions. Using a physically-based rendering engine, we synthesize a dataset to simulate this lighting-conditioned transformation with 3D head assets under varying lighting. We propose two training and inference strategies to bridge the gap between the synthetic and real image domains: (1) multi-task training that takes advantage of real human portraits without lighting labels; (2) an inference time diffusion sampling procedure based on classifier-free guidance that leverages the input portrait to better preserve details. Our method generalizes to diverse real photographs and produces realistic illumination effects, including specular highlights and cast shadows, while preserving the subject's identity. Our quantitative experiments on Light Stage data demonstrate results comparable to state-of-the-art relighting methods. Our qualitative results on in-the-wild images showcase rich and unprecedented illumination effects. Project Page: url{https://vrroom.github.io/synthlight/}

Problem

• Portrait Photography
• Lighting Optimization
• Computer Vision

Innovation

• SynthLight
• Diffusion Model
• Lighting Simulation