Generative Landmarks Guided Eyeglasses Removal 3D Face Reconstruction

2024-12-25

Conference on Multimedia Modeling

AI Summary

This work addresses the challenging problem of single-image 3D face reconstruction for subjects wearing glasses, where occlusion by eyewear severely degrades geometric and textural fidelity. We propose the first end-to-end framework that jointly performs unsupervised glasses removal and high-fidelity 3D geometry and texture recovery—without requiring paired data or ground-truth occlusion masks. Methodologically, we introduce a generative keypoint-guided mechanism to decouple glasses-region modeling from facial geometry learning; integrate implicit neural representations (INRs) with differentiable rendering; and incorporate adversarial keypoint prediction, glasses mask distillation, and multi-scale geometric regularization. Evaluated on standard benchmarks including NoW and MICC, our approach achieves state-of-the-art performance—reducing normalized mean error (NME) by 18.7%—with significantly improved reconstruction accuracy and visual realism over prior methods, demonstrating strong generalization capability.

Technology Category

• Computer Vision: 3D Computer Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Data transparency and provenance
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Problem

• 3D Face Reconstruction
• Glasses Removal
• Partial Occlusion

Innovation

• 3D face reconstruction
• glasses removal
• deep learning