Biomechanics-aware Multi-view Markerless Motion Capture of Dexterous Hand Movements

2026-07-02

AI Summary

Existing markerless hand motion capture methods suffer from low accuracy under complex dexterous motions, are highly susceptible to occlusions, and often fail to satisfy biomechanical constraints. To address these limitations, this work proposes an end-to-end, multi-view markerless hand motion capture approach that integrates a differentiable biomechanical hand model into a gradient-based optimization pipeline, jointly refining pose and shape parameters directly from multi-view video inputs. Evaluated on an 8-camera system, the method demonstrates strong robustness to occlusions and intricate hand gestures, achieving a 100% reconstruction success rate across 121 test sequences—significantly outperforming conventional two-stage methods, which attain only 85%. The reconstructed hand motions exhibit superior physiological plausibility, adhering closely to realistic biomechanical behavior.

Abstract

Markerless motion capture (MMC) techniques have been widely beneficial in biomechanical analysis of human movement; however, application to complex motions of the hand lags other musculoskeletal systems. The primary goal of this study was to evaluate the performance of a biomechanical reconstruction method that implements a gradient-based optimization approach with a biomechanical model in the loop for tracking dexterous, unconstrained hand movements using MMC. Using a custom, 8-camera setup, we acquired 121 video recordings from 6 participants performing 11 different tasks that spanned 6 hand postures, 5 object manipulation tasks, and involved motion of the proximal upper limb joints. Performance of the proposed MMC pipeline was directly compared to a more commonly adopted two-stage reconstruction method that first triangulates 2D keypoints from computer vision pose estimation algorithms to 3D and then enforces biomechanical constraints by solving a constrained inverse kinematics problem. Relative performance was assessed qualitatively by visual inspection and quantitatively using a computer vision metric. Our method generated solutions for all 121 video recordings; the two-stage method did not converge for 15% of the recordings. Across the remaining videos, our method produced more biomechanically plausible hand kinematics than the two-stage method and was more robust to occlusion effects during tasks that involved objects. The relative robustness of the end-to-end method suggests that it is more effective in utilizing the available 2D digital keypoint information. Automatic and biomechanically meaningful tracking of hand kinematics during dexterous movements has the potential to support clinical evaluation, rehabilitation monitoring, and studies of human motor control.

Problem

• markerless motion capture
• hand kinematics
• biomechanics
• dexterous hand movements
• motion tracking

Innovation

• biomechanics-aware
• markerless motion capture
• gradient-based optimization
• hand kinematics
• end-to-end reconstruction