Monocular Avatar Reconstruction via Cascaded Diffusion Priors and UV-Space Differentiable Shading

2026-06-26

AI Summary

This work addresses the challenge of reconstructing high-fidelity, relightable 3D digital humans from a single in-the-wild image, hindered by scarce training data and the difficulty of disentangling lighting from material properties. The authors propose a unified pre-trained diffusion model framework that leverages cascaded LoRA modules in UV space to sequentially perform texture completion, de-lighting, and physically based rendering (PBR) material decomposition. To enforce physical plausibility, they introduce a cross-intrinsic attention mechanism and a differentiable BRDF shading loss. Remarkably, the method achieves state-of-the-art results using fewer than 100 real 3D scans, generating 4K-resolution PBR material maps with rich geometric detail and strong generalization capabilities—significantly outperforming existing approaches under extremely limited data conditions.

Abstract

Reconstructing high-fidelity, relightable 3D avatars from a single in-the-wild image is a challenging ill-posed problem, primarily hindered by the scarcity of high-quality PBR data and the complexity of disentangling illumination from intrinsic materials. In this paper, we present a data-efficient framework that leverages the robust priors of a unified pre-trained diffusion backbone to sequentially address texture completion, delighting, and material decomposition. Unlike existing methods that rely on fragmented pipelines or extensive proprietary datasets, we utilize cascaded Low-Rank Adaptations (LoRAs) to adapt the strong generative prior of the diffusion model for each sub-task in UV space. Specifically, we first employ an Inpainting LoRA to complete missing UV textures caused by occlusion, leveraging the model's semantic understanding to generate semantically and photometrically coherent details. Subsequently, a Light-Homogenization LoRA and a novel Cross-Intrinsic Attention mechanism are introduced to remove baked-in lighting and collaboratively synthesize pixel-aligned PBR maps (Albedo, Normal, Roughness, Specular, and Displacement). To ensure physical plausibility, we impose a UV-space differentiable BRDF shading loss during the decomposition stage, forcing the generative process to adhere to the rendering equation without the artifacts typical of rasterization-based supervision. Extensive experiments demonstrate that our method, trained on fewer than 100 real 3D scans, generates comprehensive, 4K-resolution PBR assets with superior realism and generalization compared to state-of-the-art methods, and all training code and model weights will be released upon acceptance.

Problem

• Monocular Avatar Reconstruction
• Relightable 3D Avatars
• PBR Material Decomposition
• Illumination Disentanglement
• UV-Space Shading

Innovation

• Cascaded Diffusion Priors
• UV-Space Differentiable Shading
• Low-Rank Adaptation (LoRA)
• Cross-Intrinsic Attention
• PBR Material Decomposition