JADE: Joint-aware Latent Diffusion for 3D Human Generative Modeling

2024-12-29

AI Summary

Existing 3D human generation methods struggle to simultaneously achieve model simplicity, geometric fidelity, and semantic controllability—particularly for fine-grained shape editing and physical plausibility. This paper proposes a joint-aware disentangled latent representation framework that explicitly decomposes the human body into skeletal structure (joint locations) and local surface geometry (joint-attached features), modeled via a cascaded dual-diffusion process. Its core innovation is the first joint-conditioned disentangled latent space, unifying geometric interpretability with semantic editability. Evaluated on public benchmarks including SMPL-X, our method achieves significant improvements: +3.2% Chamfer Distance (CD) in autoencoding reconstruction accuracy, +27.6% LPIPS consistency in semantic editability, and −18.4% Fréchet Inception Distance (FID) in generation quality—outperforming state-of-the-art approaches across multiple metrics.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Generative modeling of 3D human bodies have been studied extensively in computer vision. The core is to design a compact latent representation that is both expressive and semantically interpretable, yet existing approaches struggle to achieve both requirements. In this work, we introduce JADE, a generative framework that learns the variations of human shapes with fined-grained control. Our key insight is a joint-aware latent representation that decomposes human bodies into skeleton structures, modeled by joint positions, and local surface geometries, characterized by features attached to each joint. This disentangled latent space design enables geometric and semantic interpretation, facilitating users with flexible controllability. To generate coherent and plausible human shapes under our proposed decomposition, we also present a cascaded pipeline where two diffusions are employed to model the distribution of skeleton structures and local surface geometries respectively. Extensive experiments are conducted on public datasets, where we demonstrate the effectiveness of JADE framework in multiple tasks in terms of autoencoding reconstruction accuracy, editing controllability and generation quality compared with existing methods.

Problem

• 3D human model generation
• shape complexity control
• realism and understandability

Innovation

• Joint-aware modeling
• 3D human body generation
• Geometric accuracy