Relightable Full-Body Gaussian Codec Avatars
🤖 AI Summary
Existing full-body 3D Gaussian-splatting avatars struggle to achieve photorealistic and temporally consistent relighting under pose variations.
Method: We propose a local–nonlocal light transport decoupling framework: (i) learnable zonal harmonics model pose-invariant local diffuse reflectance; (ii) a shadow network predicts nonlocal inter-part occlusion; and (iii) deferred shading models specular reflection and eye corneal highlights. Our approach innovatively integrates zonal harmonics rotation, precomputed incident irradiance-driven shadow estimation, and the Gaussian splatting avatar architecture.
Results: Experiments demonstrate strong generalization to novel lighting conditions and unseen poses. To our knowledge, ours is the first method enabling real-time, high-fidelity relighting of full-body avatars—including physically plausible eyeball reflections—while significantly outperforming state-of-the-art approaches in both visual quality and geometric consistency.
Application Category
📝 Abstract
We propose Relightable Full-Body Gaussian Codec Avatars, a new approach for modeling relightable full-body avatars with fine-grained details including face and hands. The unique challenge for relighting full-body avatars lies in the large deformations caused by body articulation and the resulting impact on appearance caused by light transport. Changes in body pose can dramatically change the orientation of body surfaces with respect to lights, resulting in both local appearance changes due to changes in local light transport functions, as well as non-local changes due to occlusion between body parts. To address this, we decompose the light transport into local and non-local effects. Local appearance changes are modeled using learnable zonal harmonics for diffuse radiance transfer. Unlike spherical harmonics, zonal harmonics are highly efficient to rotate under articulation. This allows us to learn diffuse radiance transfer in a local coordinate frame, which disentangles the local radiance transfer from the articulation of the body. To account for non-local appearance changes, we introduce a shadow network that predicts shadows given precomputed incoming irradiance on a base mesh. This facilitates the learning of non-local shadowing between the body parts. Finally, we use a deferred shading approach to model specular radiance transfer and better capture reflections and highlights such as eye glints. We demonstrate that our approach successfully models both the local and non-local light transport required for relightable full-body avatars, with a superior generalization ability under novel illumination conditions and unseen poses.
Problem
Research questions and friction points this paper is trying to address.
3D Character Modeling
Dynamic Lighting Adjustment
Pose-Dependent Reflections
Innovation
Methods, ideas, or system contributions that make the work stand out.
Harmonic Processing
Shadow Network
Deferred Shading
🔎 Similar Papers
No similar papers found.
