Hybrid Explicit Representation for Ultra-Realistic Head Avatars

2024-03-18

AI Summary

To address the challenge of real-time rendering (≥30 fps at 2048×1334) for photorealistic head avatars, this work proposes the first explicit hybrid representation framework integrating UV-mapped meshes with 3D Gaussian splatting. Methodologically, Gaussian point sets are rigidly bound to mesh patches and jointly optimized with texture maps, opacity maps, and geometry; a stable hybrid depth-sorting strategy is introduced to eliminate inter-patch z-buffer artifacts; and parametric tracking driven by multi-view RGB video is coupled with alpha-blended rendering. Compared to state-of-the-art methods, our approach achieves significant improvements in facial geometric detail, texture fidelity, lighting consistency, and dynamic expressiveness—while maintaining real-time performance. To our knowledge, this is the first method enabling high-resolution head avatar rendering that simultaneously delivers both photorealism and computational efficiency.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

We introduce a novel approach to creating ultra-realistic head avatars and rendering them in real-time (>30fps at $2048 imes 1334$ resolution). First, we propose a hybrid explicit representation that combines the advantages of two primitive-based efficient rendering techniques. UV-mapped 3D mesh is utilized to capture sharp and rich textures on smooth surfaces, while 3D Gaussian Splatting is employed to represent complex geometric structures. In the pipeline of modeling an avatar, after tracking parametric models based on captured multi-view RGB videos, our goal is to simultaneously optimize the texture and opacity map of mesh, as well as a set of 3D Gaussian splats localized and rigged onto the mesh facets. Specifically, we perform $alpha$-blending on the color and opacity values based on the merged and re-ordered z-buffer from the rasterization results of mesh and 3DGS. This process involves the mesh and 3DGS adaptively fitting the captured visual information to outline a high-fidelity digital avatar. To avoid artifacts caused by Gaussian splats crossing the mesh facets, we design a stable hybrid depth sorting strategy. Experiments illustrate that our modeled results exceed those of state-of-the-art approaches.

Problem

• Create ultra-realistic head avatars
• Render avatars in real-time
• Optimize texture and opacity maps

Innovation

• Hybrid explicit representation for avatars
• UV-mapped 3D mesh for textures
• 3D Gaussian Splatting for geometry