Surfel-based Gaussian Inverse Rendering for Fast and Relightable Dynamic Human Reconstruction from Monocular Video

2024-07-21

IEEE Transactions on Pattern Analysis and Machine Intelligence

AI Summary

This work addresses the challenge of efficient, relightable reconstruction of dynamic, clothed humans from monocular video. Methodologically, we propose the first surfel-based Gaussian inverse rendering framework, introducing an occlusion approximation strategy and a progressive training scheme to decouple material and illumination estimation. Our approach integrates pre-integrated lighting computation, image-based lighting (IBL), physically based rendering (PBR) material modeling, and a hybrid surfel-Gaussian representation—achieving high geometric fidelity and photorealistic relighting while preserving physical plausibility. Compared to implicit representations, our method enables real-time rendering, supports high-fidelity dynamic avatars under novel poses and arbitrary lighting conditions, and achieves a synergistic breakthrough in efficiency, visual realism, and generalization.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Efficient and accurate reconstruction of a relightable, dynamic clothed human avatar from a monocular video is crucial for the entertainment industry. This paper presents SGIA (Surfel-based Gaussian Inverse Avatar), which introduces efficient training and rendering for relightable dynamic human reconstruction. SGIA advances previous Gaussian Avatar methods by comprehensively modeling Physically-Based Rendering (PBR) properties for clothed human avatars, allowing for the manipulation of avatars into novel poses under diverse lighting conditions. Specifically, our approach integrates pre-integration and image-based lighting for fast light calculations that surpass the performance of existing implicit-based techniques. To address challenges related to material lighting disentanglement and accurate geometry reconstruction, we propose an innovative occlusion approximation strategy and a progressive training approach. Extensive experiments demonstrate that SGIA not only achieves highly accurate physical properties but also significantly enhances the realistic relighting of dynamic human avatars, providing a substantial speed advantage. We exhibit more results in our project page: https://GS-IA.github.io.

Problem

• Reconstructing relightable dynamic human avatars from monocular video efficiently
• Modeling physically-based rendering properties for clothed human avatar manipulation
• Addressing material lighting disentanglement and accurate geometry reconstruction challenges

Innovation

• Surfel-based Gaussian modeling with PBR properties
• Pre-integration and image-based lighting for speed
• Occlusion approximation and progressive training approach