BecomingLit: Relightable Gaussian Avatars with Hybrid Neural Shading

2025-06-06

AI Summary

Existing avatar models suffer from poor relighting quality under dynamic lighting and facial expressions, alongside low rendering efficiency. To address this, we propose the first lightweight, animatable, and material-geometry decoupled Gaussian radiance field (GRF) framework for high-fidelity head avatars. Methodologically, we design a low-cost multi-light facial capture system and construct a large-scale dynamic lighting dataset; employ FLAME-guided 3D Gaussian primitives; integrate an expression-dependent dynamics network with hybrid neural shading (neural diffuse BRDF + analytical specular term) to enable full-spectrum relighting (point lights + environment lighting); and unify dynamic light-field acquisition, GRF reconstruction, and monocular video-driven animation. Experiments on our benchmark demonstrate significant improvements over state-of-the-art methods, enabling real-time novel-view synthesis at >30 FPS, cross-video facial reenactment, and robust generalization to unseen lighting conditions and expressions.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce BecomingLit, a novel method for reconstructing relightable, high-resolution head avatars that can be rendered from novel viewpoints at interactive rates. Therefore, we propose a new low-cost light stage capture setup, tailored specifically towards capturing faces. Using this setup, we collect a novel dataset consisting of diverse multi-view sequences of numerous subjects under varying illumination conditions and facial expressions. By leveraging our new dataset, we introduce a new relightable avatar representation based on 3D Gaussian primitives that we animate with a parametric head model and an expression-dependent dynamics module. We propose a new hybrid neural shading approach, combining a neural diffuse BRDF with an analytical specular term. Our method reconstructs disentangled materials from our dynamic light stage recordings and enables all-frequency relighting of our avatars with both point lights and environment maps. In addition, our avatars can easily be animated and controlled from monocular videos. We validate our approach in extensive experiments on our dataset, where we consistently outperform existing state-of-the-art methods in relighting and reenactment by a significant margin.

Problem

• Reconstruct relightable high-resolution head avatars
• Develop low-cost light stage for face capture
• Create hybrid neural shading for realistic avatar rendering

Innovation

• Low-cost light stage capture setup
• Hybrid neural shading approach
• 3D Gaussian primitives with animation