Reconstructing Humans with a Biomechanically Accurate Skeleton

2025-03-27

AI Summary

This work addresses performance degradation in single-image 3D human reconstruction under extreme poses or viewpoints, and unnatural joint rotations violating biomechanical constraints. We propose a biomechanics-driven reconstruction framework. Methodologically: (1) we construct an anatomically grounded skeletal model that explicitly encodes physiological limits on joint angles; (2) we employ a Transformer architecture to directly regress SMPL-X parameters; (3) we introduce a pseudo-labeling iterative optimization scheme to mitigate the scarcity of ground-truth 3D annotations. Our key contributions are: the first integration of kinematic joint constraints into an end-to-end reconstruction paradigm; competitive performance on state-of-the-art benchmarks, with significant accuracy improvements under extreme poses and viewpoints; and strict adherence of all joint rotations to anatomical feasibility. The code, models, and datasets are fully open-sourced.

Technology Category

• Computer Vision: 3D Computer Vision
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Machine Learning: Feature Construction/Reformulation

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

In this paper, we introduce a method for reconstructing 3D humans from a single image using a biomechanically accurate skeleton model. To achieve this, we train a transformer that takes an image as input and estimates the parameters of the model. Due to the lack of training data for this task, we build a pipeline to produce pseudo ground truth model parameters for single images and implement a training procedure that iteratively refines these pseudo labels. Compared to state-of-the-art methods for 3D human mesh recovery, our model achieves competitive performance on standard benchmarks, while it significantly outperforms them in settings with extreme 3D poses and viewpoints. Additionally, we show that previous reconstruction methods frequently violate joint angle limits, leading to unnatural rotations. In contrast, our approach leverages the biomechanically plausible degrees of freedom making more realistic joint rotation estimates. We validate our approach across multiple human pose estimation benchmarks. We make the code, models and data available at: https://isshikihugh.github.io/HSMR/

Problem

• Reconstructing 3D humans from single images accurately
• Addressing lack of training data with pseudo labels
• Ensuring biomechanically plausible joint rotations

Innovation

• Transformer estimates biomechanical skeleton parameters
• Pipeline generates pseudo ground truth labels
• Biomechanical constraints ensure realistic joint rotations