3DPR: Single Image 3D Portrait Relight using Generative Priors

2025-10-17

AI Summary

This paper addresses the ill-posed problem of 3D relighting from a single portrait image. We propose a lightweight relighting framework based on a latent-space triplane reflectance network. Methodologically, it jointly models high-fidelity geometry, BRDF material properties, and HDR environment lighting by integrating latent priors from pre-trained generative models, encoder-based inverse embedding, and multi-view 4K OLAT light-field data. Crucially, we introduce an in-the-wild image-driven generative prior for facial geometry and employ a triplane representation to enable efficient, differentiable reflectance rendering. Unlike conventional differentiable renderers, our approach decouples geometry and material modeling, thereby overcoming inherent coupling limitations. Quantitative and qualitative evaluations demonstrate significant improvements in identity preservation and accurate reproduction of complex lighting phenomena—including specular highlights, self-shadowing, and subsurface scattering.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Rendering novel, relit views of a human head, given a monocular portrait image as input, is an inherently underconstrained problem. The traditional graphics solution is to explicitly decompose the input image into geometry, material and lighting via differentiable rendering; but this is constrained by the multiple assumptions and approximations of the underlying models and parameterizations of these scene components. We propose 3DPR, an image-based relighting model that leverages generative priors learnt from multi-view One-Light-at-A-Time (OLAT) images captured in a light stage. We introduce a new diverse and large-scale multi-view 4K OLAT dataset of 139 subjects to learn a high-quality prior over the distribution of high-frequency face reflectance. We leverage the latent space of a pre-trained generative head model that provides a rich prior over face geometry learnt from in-the-wild image datasets. The input portrait is first embedded in the latent manifold of such a model through an encoder-based inversion process. Then a novel triplane-based reflectance network trained on our lightstage data is used to synthesize high-fidelity OLAT images to enable image-based relighting. Our reflectance network operates in the latent space of the generative head model, crucially enabling a relatively small number of lightstage images to train the reflectance model. Combining the generated OLATs according to a given HDRI environment maps yields physically accurate environmental relighting results. Through quantitative and qualitative evaluations, we demonstrate that 3DPR outperforms previous methods, particularly in preserving identity and in capturing lighting effects such as specularities, self-shadows, and subsurface scattering. Project Page: https://vcai.mpi-inf.mpg.de/projects/3dpr/

Problem

• Rendering relit 3D portraits from single monocular images
• Overcoming limitations of traditional geometry-material-light decomposition
• Generating physically accurate lighting with identity preservation

Innovation

• Leverages generative priors from multi-view OLAT images
• Uses latent space of pre-trained generative head model
• Employs triplane-based reflectance network for relighting