Arc2Avatar: Generating Expressive 3D Avatars from a Single Image via ID Guidance

2025-01-09

AI Summary

This work addresses the problem of reconstructing high-fidelity, expression-editable 3D avatars from a single facial image. We propose the first identity (ID)-guided single-image Score Distillation Sampling (SDS) framework, integrating a fine-tuned face foundation model with an enhanced 3D Gaussian Splatting representation. To ensure geometric consistency, we introduce a dense facial template mapping; additionally, we design a connectivity regularization module and an expression correction module to enable precise blendshape-driven animation and photorealistic facial expression editing. Our method achieves state-of-the-art performance in both ID preservation and visual fidelity, significantly alleviating color distortion commonly observed in prior approaches. It enables fine-grained expression control and supports diverse, semantically meaningful emotional expression generation—demonstrating superior controllability and realism compared to existing single-image 3D face reconstruction methods.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Inspired by the effectiveness of 3D Gaussian Splatting (3DGS) in reconstructing detailed 3D scenes within multi-view setups and the emergence of large 2D human foundation models, we introduce Arc2Avatar, the first SDS-based method utilizing a human face foundation model as guidance with just a single image as input. To achieve that, we extend such a model for diverse-view human head generation by fine-tuning on synthetic data and modifying its conditioning. Our avatars maintain a dense correspondence with a human face mesh template, allowing blendshape-based expression generation. This is achieved through a modified 3DGS approach, connectivity regularizers, and a strategic initialization tailored for our task. Additionally, we propose an optional efficient SDS-based correction step to refine the blendshape expressions, enhancing realism and diversity. Experiments demonstrate that Arc2Avatar achieves state-of-the-art realism and identity preservation, effectively addressing color issues by allowing the use of very low guidance, enabled by our strong identity prior and initialization strategy, without compromising detail.

Problem

• 3D character generation
• photo-to-3D conversion
• emotion expression

Innovation

• Arc2Avatar
• 3D Gaussian Spraying
• Realistic Avatar Generation