Refined Geometry-guided Head Avatar Reconstruction from Monocular RGB Video

2025-03-27

AI Summary

High-fidelity monocular RGB video-driven head reconstruction—especially for complex facial details—remains challenging due to the geometric limitations of 3D Morphable Models (3DMMs). Method: We propose a two-stage differentiable framework. Stage I jointly optimizes a Neural Radiance Field (NeRF) and frame-shared latent codes under 3DMM priors. Stage II initializes a Signed Distance Function (SDF) from the NeRF density field and refines a mesh via a Laplacian-smoothed displacement field, overcoming 3DMM’s geometric constraints. Contribution/Results: This is the first work to integrate SDF-guided mesh refinement with geometry-aware Laplacian-smoothed displacement fields for monocular head reconstruction without depth or multi-view supervision. Our method achieves geometrically consistent, detail-rich, and temporally coherent reconstructions. Experiments demonstrate state-of-the-art performance in rendering quality, geometric accuracy, and fine-grained expression modeling—including lip motion and micro-expressions.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

High-fidelity reconstruction of head avatars from monocular videos is highly desirable for virtual human applications, but it remains a challenge in the fields of computer graphics and computer vision. In this paper, we propose a two-phase head avatar reconstruction network that incorporates a refined 3D mesh representation. Our approach, in contrast to existing methods that rely on coarse template-based 3D representations derived from 3DMM, aims to learn a refined mesh representation suitable for a NeRF that captures complex facial nuances. In the first phase, we train 3DMM-stored NeRF with an initial mesh to utilize geometric priors and integrate observations across frames using a consistent set of latent codes. In the second phase, we leverage a novel mesh refinement procedure based on an SDF constructed from the density field of the initial NeRF. To mitigate the typical noise in the NeRF density field without compromising the features of the 3DMM, we employ Laplace smoothing on the displacement field. Subsequently, we apply a second-phase training with these refined meshes, directing the learning process of the network towards capturing intricate facial details. Our experiments demonstrate that our method further enhances the NeRF rendering based on the initial mesh and achieves performance superior to state-of-the-art methods in reconstructing high-fidelity head avatars with such input.

Problem

• Reconstruct high-fidelity head avatars from monocular videos
• Learn refined mesh representation for NeRF capturing facial nuances
• Enhance NeRF rendering with refined meshes and geometric priors

Innovation

• Two-phase head avatar reconstruction network
• Refined 3D mesh representation for NeRF
• Laplace smoothing on displacement field