SAM 3D Body: Robust Full-Body Human Mesh Recovery

2026-02-17

AI Summary

This work addresses the limited generalization and inconsistent accuracy of single-image full-body 3D human reconstruction in complex in-the-wild scenes. To this end, we propose an encoder-decoder framework that supports multimodal user-provided cues—such as 2D keypoints and segmentation masks—to enable high-fidelity, user-guided reconstruction of full-body pose and shape, including hands and feet. Our key innovation lies in the introduction of the Momentum Human Representation (MHR), a novel parametric model that decouples skeletal structure from surface geometry. We further develop a multi-stage annotation pipeline integrating manual labeling, differentiable optimization, multi-view geometry, and dense keypoints, coupled with an efficient data curation strategy. Extensive experiments demonstrate that our method significantly outperforms existing approaches across diverse real-world scenarios, achieving state-of-the-art performance in both quantitative metrics and user preference studies. The model and MHR representation are publicly released.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We introduce SAM 3D Body (3DB), a promptable model for single-image full-body 3D human mesh recovery (HMR) that demonstrates state-of-the-art performance, with strong generalization and consistent accuracy in diverse in-the-wild conditions. 3DB estimates the human pose of the body, feet, and hands. It is the first model to use a new parametric mesh representation, Momentum Human Rig (MHR), which decouples skeletal structure and surface shape. 3DB employs an encoder-decoder architecture and supports auxiliary prompts, including 2D keypoints and masks, enabling user-guided inference similar to the SAM family of models. We derive high-quality annotations from a multi-stage annotation pipeline that uses various combinations of manual keypoint annotation, differentiable optimization, multi-view geometry, and dense keypoint detection. Our data engine efficiently selects and processes data to ensure data diversity, collecting unusual poses and rare imaging conditions. We present a new evaluation dataset organized by pose and appearance categories, enabling nuanced analysis of model behavior. Our experiments demonstrate superior generalization and substantial improvements over prior methods in both qualitative user preference studies and traditional quantitative analysis. Both 3DB and MHR are open-source.

Problem

• 3D human mesh recovery
• full-body reconstruction
• in-the-wild conditions
• pose estimation
• generalization

Innovation

• promptable model
• Momentum Human Rig
• 3D human mesh recovery
• auxiliary prompts
• multi-stage annotation