FreeUV: Ground-Truth-Free Realistic Facial UV Texture Recovery via Cross-Assembly Inference Strategy

2025-03-21

AI Summary

This work addresses the challenging problem of unsupervised high-fidelity UV texture reconstruction from single-view 2D face images—particularly under complex makeup, wrinkles, and occlusions, where fine details are prone to distortion. We propose the first end-to-end framework that requires no ground-truth or synthetic UV annotations. Methodologically, we build upon a pretrained Stable Diffusion model and introduce a dual-branch architecture: branches are trained independently but assembled synergistically during inference. We incorporate implicit geometric guidance and explicit texture priors, and pioneer a cross-assembly inference strategy that decouples appearance fidelity from structural consistency. Quantitative and qualitative evaluations demonstrate state-of-the-art performance across multiple benchmarks. Moreover, our method supports localized editing, latent-space feature interpolation, and multi-view texture generation—significantly enhancing generalization to real-world scenarios.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Recovering high-quality 3D facial textures from single-view 2D images is a challenging task, especially under constraints of limited data and complex facial details such as makeup, wrinkles, and occlusions. In this paper, we introduce FreeUV, a novel ground-truth-free UV texture recovery framework that eliminates the need for annotated or synthetic UV data. FreeUV leverages pre-trained stable diffusion model alongside a Cross-Assembly inference strategy to fulfill this objective. In FreeUV, separate networks are trained independently to focus on realistic appearance and structural consistency, and these networks are combined during inference to generate coherent textures. Our approach accurately captures intricate facial features and demonstrates robust performance across diverse poses and occlusions. Extensive experiments validate FreeUV's effectiveness, with results surpassing state-of-the-art methods in both quantitative and qualitative metrics. Additionally, FreeUV enables new applications, including local editing, facial feature interpolation, and multi-view texture recovery. By reducing data requirements, FreeUV offers a scalable solution for generating high-fidelity 3D facial textures suitable for real-world scenarios.

Problem

• Recovering 3D facial textures from single 2D images without ground-truth data
• Handling complex facial details like makeup and occlusions robustly
• Reducing data dependency for scalable high-fidelity texture generation

Innovation

• Ground-truth-free UV texture recovery framework
• Cross-Assembly inference strategy
• Leverages pre-trained stable diffusion model