PHD: Personalized 3D Human Body Fitting with Point Diffusion

2025-08-28

AI Summary

Traditional 3D human mesh recovery methods neglect inter-subject anatomical variability and rely solely on 2D image constraints, leading to inaccurate 3D pose estimation and poor pelvis localization. This paper proposes PHD, a user-specific two-stage fitting framework: first calibrating subject-specific body shape, then decoupling and optimizing 3D pose conditioned on the calibrated shape. Our key innovation is the explicit coupling of personalized shape modeling with a learned 3D pose prior—implemented via a Point Diffusion Transformer—and the introduction of a Point Distillation Sampling loss, enabling iterative optimization using synthetic data only. PHD is plug-and-play, significantly improving absolute pose accuracy and pelvis alignment precision. While maintaining high data efficiency, it establishes a novel, generalizable, and personalized paradigm for 3D human mesh recovery.

Technology Category

• Computer Vision: 3D Computer Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Learning with Manifolds

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

We introduce PHD, a novel approach for personalized 3D human mesh recovery (HMR) and body fitting that leverages user-specific shape information to improve pose estimation accuracy from videos. Traditional HMR methods are designed to be user-agnostic and optimized for generalization. While these methods often refine poses using constraints derived from the 2D image to improve alignment, this process compromises 3D accuracy by failing to jointly account for person-specific body shapes and the plausibility of 3D poses. In contrast, our pipeline decouples this process by first calibrating the user's body shape and then employing a personalized pose fitting process conditioned on that shape. To achieve this, we develop a body shape-conditioned 3D pose prior, implemented as a Point Diffusion Transformer, which iteratively guides the pose fitting via a Point Distillation Sampling loss. This learned 3D pose prior effectively mitigates errors arising from an over-reliance on 2D constraints. Consequently, our approach improves not only pelvis-aligned pose accuracy but also absolute pose accuracy -- an important metric often overlooked by prior work. Furthermore, our method is highly data-efficient, requiring only synthetic data for training, and serves as a versatile plug-and-play module that can be seamlessly integrated with existing 3D pose estimators to enhance their performance. Project page: https://phd-pose.github.io/

Problem

• Personalized 3D human mesh recovery from videos
• Improving pose estimation accuracy with user-specific shape
• Mitigating errors from over-reliance on 2D constraints

Innovation

• Personalized pose fitting with user-specific shape calibration
• Shape-conditioned 3D pose prior using Point Diffusion Transformer
• Point Distillation Sampling loss for iterative pose guidance