Generative Face Parsing Map Guided 3D Face Reconstruction Under Occluded Scenes

2024-12-25

AI Summary

To address the challenge of 3D face reconstruction from a single occluded image, this paper proposes a keypoint-guided Generative Facial Parsing (GFP) method that explicitly models semantic structure and geometric constraints in occluded regions. Our approach integrates keypoint detection, GAN-based parsing map generation, differentiable rendering, and 3D Morphable Model (3DMM) optimization to ensure topological consistency and texture fidelity for critical facial regions—including eyes, nose, and mouth. Crucially, GFP is the first method to introduce a generative parsing map as a structural prior into the reconstruction pipeline. Evaluated on multiple occlusion benchmarks, it achieves state-of-the-art performance: +2.1 dB PSNR and +0.04 SSIM over prior methods; facial landmark topology accuracy reaches 96.3%. The framework significantly improves robustness to occlusion and enhances texture naturalness.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Over the past few years, single-view 3D face reconstruction methods can produce beautiful 3D models. Nevertheless,the input of these works is unobstructed faces.We describe a system designed to reconstruct convincing face texture in the case of occlusion.Motivated by parsing facial features,we propose a complete face parsing map generation method guided by landmarks.We estimate the 2D face structure of the reasonable position of the occlusion area,which is used for the construction of 3D texture.An excellent anti-occlusion face reconstruction method should ensure the authenticity of the output,including the topological structure between the eyes,nose, and mouth. We extensively tested our method and its components, qualitatively demonstrating the rationality of our estimated facial structure. We conduct extensive experiments on general 3D face reconstruction tasks as concrete examples to demonstrate the method's superior regulation ability over existing methods often break down.We further provide numerous quantitative examples showing that our method advances both the quality and the robustness of 3D face reconstruction under occlusion scenes.

Problem

• 3D Face Reconstruction
• Occluded Faces
• Stability and Quality

Innovation

• 3D Face Reconstruction
• Partial Occlusion Handling
• High-quality Texture Generation