LAGA: Layered 3D Avatar Generation and Customization via Gaussian Splatting

2024-05-21

arXiv.org

AI Summary

This work addresses the strong coupling between human body and clothing and poor editability in text-driven 3D dressed avatar generation. We propose the first hierarchical Gaussian modeling framework that decouples the body and garments into independent, editable 3D Gaussian layers. Methodologically, we adopt a coarse-to-fine garment generation strategy, introduce a dual Score Distillation Sampling (SDS) consistency loss to enforce cross-layer semantic alignment, and design triple regularization—deformation-, geometry-, and topology-aware—to enhance robustness to dynamic articulation. Leveraging hierarchical 3D Gaussian Splatting representations and diffusion-based text-to-garment generation, our approach achieves high-fidelity reconstruction. Experiments demonstrate significant improvements over state-of-the-art methods across multiple datasets, achieving breakthroughs in generation fidelity, garment-level free replacement capability, and cross-avatar garment transfer fidelity.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Creating and customizing a 3D clothed avatar from textual descriptions is a critical and challenging task. Traditional methods often treat the human body and clothing as inseparable, limiting users' ability to freely mix and match garments. In response to this limitation, we present LAyered Gaussian Avatar (LAGA), a carefully designed framework enabling the creation of high-fidelity decomposable avatars with diverse garments. By decoupling garments from avatar, our framework empowers users to conviniently edit avatars at the garment level. Our approach begins by modeling the avatar using a set of Gaussian points organized in a layered structure, where each layer corresponds to a specific garment or the human body itself. To generate high-quality garments for each layer, we introduce a coarse-to-fine strategy for diverse garment generation and a novel dual-SDS loss function to maintain coherence between the generated garments and avatar components, including the human body and other garments. Moreover, we introduce three regularization losses to guide the movement of Gaussians for garment transfer, allowing garments to be freely transferred to various avatars. Extensive experimentation demonstrates that our approach surpasses existing methods in the generation of 3D clothed humans.

Problem

• Enables creation of high-fidelity, decomposable 3D avatars from text.
• Allows free mixing and matching of garments on avatars.
• Introduces novel loss functions for garment-avatar coherence.

Innovation

• Layered Gaussian Avatar for decomposable 3D avatars
• Coarse-to-fine strategy for garment generation
• Dual-SDS loss for garment-avatar coherence