HeadCraft: Modeling High-Detail Shape Variations for Animated 3DMMs

2023-12-21

arXiv.org

AI Summary

Existing methods struggle to generate 3D human head models with both high geometric fidelity and explicit animation controllability from partial observations (e.g., depth scans). This paper introduces the first generative framework that maps semantically controllable vertex displacements onto a hand-crafted UV parameterization and models them via StyleGAN2. Built upon a disentangled parametric 3D Morphable Model—comprising geometric bases and a displacement field—it unifies geometry reconstruction, pose control, and semantic editing. The method supports unconditional generation, single-frame depth-scan fitting (achieving sub-millimeter accuracy), and localized editing. Evaluated on the NPHM dataset, it significantly outperforms state-of-the-art implicit representations—including NeRF and SDF-based approaches—in both geometric fidelity and explicit articulation control. To our knowledge, this is the first approach enabling high-quality, animatable, and editable generative 3D head modeling.

Abstract

Current advances in human head modeling allow the generation of plausible-looking 3D head models via neural representations, such as NeRFs and SDFs. Nevertheless, constructing complete high-fidelity head models with explicitly controlled animation remains an issue. Furthermore, completing the head geometry based on a partial observation, e.g., coming from a depth sensor, while preserving a high level of detail is often problematic for the existing methods. We introduce a generative model for detailed 3D head meshes on top of an articulated 3DMM, simultaneously allowing explicit animation and high-detail preservation. Our method is trained in two stages. First, we register a parametric head model with vertex displacements to each mesh of the recently introduced NPHM dataset of accurate 3D head scans. The estimated displacements are baked into a hand-crafted UV layout. Second, we train a StyleGAN model to generalize over the UV maps of displacements, which we later refer to as HeadCraft. The decomposition of the parametric model and high-quality vertex displacements allows us to animate the model and modify the regions semantically. We demonstrate the results of unconditional sampling, fitting to a scan and editing. The project page is available at https://seva100.github.io/headcraft.

Problem

• 3D head modeling
• animation
• partial information depth sensors

Innovation

• HeadCraft
• 3DMM
• Dynamic 3D Head Modeling