MixedGaussianAvatar: Realistically and Geometrically Accurate Head Avatar via Mixed 2D-3D Gaussian Splatting

2024-12-06

arXiv.org

AI Summary

To address the longstanding challenges in high-fidelity 3D head avatar reconstruction—namely the trade-off between geometric accuracy and rendering quality, low training/rendering efficiency, and surface inconsistency—this paper proposes a hybrid 2D-3D Gaussian representation: 2D Gaussians ensure precise geometry estimation, while 3D Gaussians enhance photorealistic rendering; both are parameterized and animated via FLAME mesh binding. We introduce a novel progressive training strategy: first optimizing 2D Gaussians independently, then jointly fine-tuning the hybrid representation. Leveraging differentiable rendering and an efficient splatting mechanism, our method achieves state-of-the-art performance across multiple benchmarks—reducing geometric error by 32%, improving PSNR by 2.1 dB over NeRF-based approaches, accelerating both training and inference, and enabling real-time animation driving.

Technology Category

• Computer Vision: 3D Computer Vision
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Reconstructing high-fidelity 3D head avatars is crucial in various applications such as virtual reality. The pioneering methods reconstruct realistic head avatars with Neural Radiance Fields (NeRF), which have been limited by training and rendering speed. Recent methods based on 3D Gaussian Splatting (3DGS) significantly improve the efficiency of training and rendering. However, the surface inconsistency of 3DGS results in subpar geometric accuracy; later, 2DGS uses 2D surfels to enhance geometric accuracy at the expense of rendering fidelity. To leverage the benefits of both 2DGS and 3DGS, we propose a novel method named MixedGaussianAvatar for realistically and geometrically accurate head avatar reconstruction. Our main idea is to utilize 2D Gaussians to reconstruct the surface of the 3D head, ensuring geometric accuracy. We attach the 2D Gaussians to the triangular mesh of the FLAME model and connect additional 3D Gaussians to those 2D Gaussians where the rendering quality of 2DGS is inadequate, creating a mixed 2D-3D Gaussian representation. These 2D-3D Gaussians can then be animated using FLAME parameters. We further introduce a progressive training strategy that first trains the 2D Gaussians and then fine-tunes the mixed 2D-3D Gaussians. We demonstrate the superiority of MixedGaussianAvatar through comprehensive experiments. The code will be released at: https://github.com/ChenVoid/MGA/.

Problem

• Improving geometric accuracy in 3D head avatar reconstruction
• Balancing rendering fidelity with computational efficiency
• Integrating 2D and 3D Gaussian representations for animation

Innovation

• Combines 2D and 3D Gaussians for accurate head avatars
• Attaches Gaussians to FLAME model mesh for animation
• Uses progressive training strategy for optimization