Learning Topology Uniformed Face Mesh by Volume Rendering for Multi-view Reconstruction

2024-04-08

arXiv.org

AI Summary

To address topological inconsistency, reliance on manual intervention, and specialized capture hardware in multi-view face 3D reconstruction, this paper proposes an end-to-end differentiable mesh optimization framework. Our method employs a feature diffusion module to propagate sparse mesh features into a continuous radiance field, enabling joint optimization of geometry and implicit appearance. A topology-preserving parameterization ensures the output is a consistent, editable triangular mesh. By integrating multi-view image supervision with neural volumetric rendering, the framework supports deformation-invariant real-time ray tracing. Evaluated on standard multi-view face datasets, our approach significantly outperforms conventional MVS-plus-registration pipelines, achieving state-of-the-art performance in both geometric accuracy and texture fidelity. Moreover, it enables real-time high-quality rendering and interactive geometric editing under animation-driven deformations.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multi-instance/Multi-view Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Face meshes in consistent topology serve as the foundation for many face-related applications, such as 3DMM constrained face reconstruction and expression retargeting. Traditional methods commonly acquire topology uniformed face meshes by two separate steps: multi-view stereo (MVS) to reconstruct shapes followed by non-rigid registration to align topology, but struggles with handling noise and non-lambertian surfaces. Recently neural volume rendering techniques have been rapidly evolved and shown great advantages in 3D reconstruction or novel view synthesis. Our goal is to leverage the superiority of neural volume rendering into multi-view reconstruction of face mesh with consistent topology. We propose a mesh volume rendering method that enables directly optimizing mesh geometry while preserving topology, and learning implicit features to model complex facial appearance from multi-view images. The key innovation lies in spreading sparse mesh features into the surrounding space to simulate radiance field required for volume rendering, which facilitates backpropagation of gradients from images to mesh geometry and implicit appearance features. Our proposed feature spreading module exhibits deformation invariance, enabling photorealistic rendering seamlessly after mesh editing. We conduct experiments on multi-view face image dataset to evaluate the reconstruction and implement an application for photorealistic rendering of animated face mesh.

Problem

• Reconstructing topology-consistent face meshes from multi-view images
• Optimizing geometry while preserving template mesh topology
• Combining explicit mesh representation with neural volume rendering

Innovation

• Combines explicit mesh with neural volume rendering
• Derives density fields using distance field intermediary
• Encodes radiance field in compact tri-planes