Generative Relightable Avatars

2026-06-21

AI Summary

This work proposes a relightable full-body digital human rendering method that addresses the one-to-many mapping challenge between appearance details and pose-dependent textures under arbitrary viewpoints. The approach integrates physics-based material optimization in UV space with generative refinement: it first estimates material parameters using a tracked mesh to produce a coarse relit output, then enhances texture details via a feedforward network, and finally employs a fine-tuned video-to-video diffusion model to generate high-fidelity, temporally coherent dynamic sequences. An error-looping strategy is introduced to support long-sequence generation. By preserving 3D lighting controllability while significantly improving visual realism, the method achieves superior perceptual quality compared to existing relightable digital human techniques.

Abstract

We present Generative Relightable Avatars (GRA), a person-specific method for photorealistic free-view rendering and environment-map relighting of full-body humans. We postulate that modeling fine-grained appearance details is inherently a one-to-many problem that can benefit from a generative formulation. In contrast to fully regressive relightable avatar methods, GRA follows a hybrid approach that combines controllable, physics-grounded relighting with probabilistic refinement. Starting from a tracked animated mesh, we optimize material parameters in UV-space and render a coarse relit appearance under a target HDR environment map. Next, we refine the textures with a feed-forward model to capture pose-dependent texture dynamics and illumination effects beyond simplified reflectance assumptions. Finally, a fine-tuned video-to-video diffusion model transforms the physically grounded renderings into temporally coherent, high-detail videos while preserving 3D control, with an error-recycling strategy for generating long videos. Experimental evaluations demonstrate our method's improved perceptual quality over prior relightable avatar baselines. Project Page: https://vcai.mpi-inf.mpg.de/projects/GRA/

Problem

• Relightable Avatars
• Photorealistic Rendering
• Full-body Humans
• Appearance Modeling
• Environment Map Relighting

Innovation

• Generative Relightable Avatars
• physically-based relighting
• pose-dependent texture refinement
• video-to-video diffusion
• temporal coherence