LUCAS: Layered Universal Codec Avatars

2025-02-27

AI Summary

Addressing the challenges of modeling dynamic face-hair interactions and poor cross-identity generalization in 3D head reconstruction, this paper proposes a hierarchical universal prior model. It decouples the hairless head and hair into two independent neural branches: the former is represented via a differentiable mesh, while the latter is modeled using a hierarchical neural radiance field (NeRF), augmented with dynamic anchor-based geometric optimization. This work achieves the first structural disentanglement of face and hair, enabling dual rendering pipelines—real-time rasterization and Gaussian splatting. Under zero-shot driving, the method significantly improves robustness in pose, expression, and hairstyle transfer. In subject-retention evaluation, it achieves +1.8 dB PSNR and +0.04 SSIM over baselines, with more natural and stable dynamic motion. Comprehensive quantitative and qualitative evaluations demonstrate consistent superiority over single-mesh and baseline Gaussian-based approaches.

Technology Category

• Computer Vision: 3D Computer Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Photorealistic 3D head avatar reconstruction faces critical challenges in modeling dynamic face-hair interactions and achieving cross-identity generalization, particularly during expressions and head movements. We present LUCAS, a novel Universal Prior Model (UPM) for codec avatar modeling that disentangles face and hair through a layered representation. Unlike previous UPMs that treat hair as an integral part of the head, our approach separates the modeling of the hairless head and hair into distinct branches. LUCAS is the first to introduce a mesh-based UPM, facilitating real-time rendering on devices. Our layered representation also improves the anchor geometry for precise and visually appealing Gaussian renderings. Experimental results indicate that LUCAS outperforms existing single-mesh and Gaussian-based avatar models in both quantitative and qualitative assessments, including evaluations on held-out subjects in zero-shot driving scenarios. LUCAS demonstrates superior dynamic performance in managing head pose changes, expression transfer, and hairstyle variations, thereby advancing the state-of-the-art in 3D head avatar reconstruction.

Problem

• Dynamic face-hair interaction modeling challenges
• Cross-identity generalization during expressions
• Real-time 3D head avatar rendering

Innovation

• Layered Universal Codec Avatars
• Separates hair and head modeling
• Mesh-based real-time rendering