Multi-Rigid-Body Approximation of Human Hands with Application to Digital Twin

2025-12-08

AI Summary

Modeling human hands in digital twins requires balancing anatomical fidelity with real-time physical simulation—a longstanding challenge. Method: This paper proposes a personalized multi-rigid-body hand modeling framework: (1) fitting subject-specific MANO models to optical motion capture data and mapping them to anatomically consistent URDF representations; (2) introducing a novel iterative projection method that combines closed-form SO(3) rotation solutions with Baker–Campbell–Hausdorff (BCH) formula-based corrections to efficiently project rotations onto physiologically valid joint constraint spaces using Lie group algebra. Results: Experiments show hand reconstruction error < 1 cm; the resulting models enable high-fidelity, real-time physics simulation. Reinforcement learning policies trained on these models successfully reproduce diverse human grasping behaviors, demonstrating effectiveness and generalization capability in digital twin interaction tasks.

Technology Category

• Intelligent Robots: Manipulation
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Computer Vision: Low Level & Physics-based Vision

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Web Mining and Content Analysis: Web data generation and simulation

Abstract

Human hand simulation plays a critical role in digital twin applications, requiring models that balance anatomical fidelity with computational efficiency. We present a complete pipeline for constructing multi-rigid-body approximations of human hands that preserve realistic appearance while enabling real-time physics simulation. Starting from optical motion capture of a specific human hand, we construct a personalized MANO (Multi-Abstracted hand model with Neural Operations) model and convert it to a URDF (Unified Robot Description Format) representation with anatomically consistent joint axes. The key technical challenge is projecting MANO's unconstrained SO(3) joint rotations onto the kinematically constrained joints of the rigid-body model. We derive closed-form solutions for single degree-of-freedom joints and introduce a Baker-Campbell-Hausdorff (BCH)-corrected iterative method for two degree-of-freedom joints that properly handles the non-commutativity of rotations. We validate our approach through digital twin experiments where reinforcement learning policies control the multi-rigid-body hand to replay captured human demonstrations. Quantitative evaluation shows sub-centimeter reconstruction error and successful grasp execution across diverse manipulation tasks.

Problem

• Develop a pipeline for multi-rigid-body hand models from motion capture
• Project unconstrained joint rotations onto kinematically constrained rigid-body joints
• Enable real-time physics simulation for digital twin hand applications

Innovation

• Personalized MANO model conversion to URDF representation
• Closed-form solutions for single degree-of-freedom joint rotations
• BCH-corrected iterative method for two degree-of-freedom joints