Enhancing Hands in 3D Whole-Body Pose Estimation with Conditional Hands Modulator

2026-03-15

AI Summary

This work addresses the challenge in 3D whole-body pose estimation where existing methods struggle to accurately model fine-grained finger poses due to insufficient hand data diversity, while specialized hand models lack global body context. To bridge this gap, the authors propose Hand4Whole++, a framework featuring a lightweight Conditional Hand Attention Modulator (CHAM) that refines whole-body feature representations through hand-conditioned modulation—without retraining the pretrained whole-body model. By integrating a differentiable rigid alignment mechanism, the approach enables joint optimization of local hand details and global body structure. This synergy significantly improves hand pose accuracy while enhancing overall 3D pose consistency, yielding notable gains across multiple benchmark metrics.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Accurately recovering hand poses within the body context remains a major challenge in 3D whole-body pose estimation. This difficulty arises from a fundamental supervision gap: whole-body pose estimators are trained on full-body datasets with limited hand diversity, while hand-only estimators, trained on hand-centric datasets, excel at detailed finger articulation but lack global body awareness. To address this, we propose Hand4Whole++, a modular framework that leverages the strengths of both pre-trained whole-body and hand pose estimators. We introduce CHAM (Conditional Hands Modulator), a lightweight module that modulates the whole-body feature stream using hand-specific features extracted from a pre-trained hand pose estimator. This modulation enables the whole-body model to predict wrist orientations that are both accurate and coherent with the upper-body kinematic structure, without retraining the full-body model. In parallel, we directly incorporate finger articulations and hand shapes predicted by the hand pose estimator, aligning them to the full-body mesh via differentiable rigid alignment. This design allows Hand4Whole++ to combine globally consistent body reasoning with fine-grained hand detail. Extensive experiments demonstrate that Hand4Whole++ substantially improves hand accuracy and enhances overall full-body pose quality.

Problem

• 3D whole-body pose estimation
• hand pose estimation
• hand articulation
• supervision gap
• body-hand coherence

Innovation

• Conditional Hands Modulator
• 3D whole-body pose estimation
• hand pose refinement
• modular fusion
• differentiable rigid alignment