FSGlove: An Inertial-Based Hand Tracking System with Shape-Aware Calibration

2025-09-25

AI Summary

Existing hand motion capture systems suffer from low degrees of freedom (≤21 DOF) and neglect inter-subject anatomical variability, limiting their applicability to contact-intensive tasks. This paper introduces the first unified optimization framework integrating 48-DOF inertial sensing with a differentiable calibration method, DiffHCal, enabling joint estimation of articulation, subject-specific hand shape parameters, and sensor biases. Leveraging an IMU array and the MANO hand model, the system achieves high-fidelity motion capture and geometric reconstruction. Experiments demonstrate joint angle errors <2.7° and significantly improved reconstruction accuracy for hand shape and fine contact details (e.g., fingertip friction) over commercial solutions. The system’s open-source hardware and software support applications in VR, robotics, and biomechanics. Key contributions include: (i) unifying high-DOF inertial perception with differentiable shape-motion co-calibration; and (ii) achieving millimeter-level contact fidelity and personalized hand modeling without external markers—previously unattained.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Learning with Manifolds

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Accurate hand motion capture (MoCap) is vital for applications in robotics, virtual reality, and biomechanics, yet existing systems face limitations in capturing high-degree-of-freedom (DoF) joint kinematics and personalized hand shape. Commercial gloves offer up to 21 DoFs, which are insufficient for complex manipulations while neglecting shape variations that are critical for contact-rich tasks. We present FSGlove, an inertial-based system that simultaneously tracks up to 48 DoFs and reconstructs personalized hand shapes via DiffHCal, a novel calibration method. Each finger joint and the dorsum are equipped with IMUs, enabling high-resolution motion sensing. DiffHCal integrates with the parametric MANO model through differentiable optimization, resolving joint kinematics, shape parameters, and sensor misalignment during a single streamlined calibration. The system achieves state-of-the-art accuracy, with joint angle errors of less than 2.7 degree, and outperforms commercial alternatives in shape reconstruction and contact fidelity. FSGlove's open-source hardware and software design ensures compatibility with current VR and robotics ecosystems, while its ability to capture subtle motions (e.g., fingertip rubbing) bridges the gap between human dexterity and robotic imitation. Evaluated against Nokov optical MoCap, FSGlove advances hand tracking by unifying the kinematic and contact fidelity. Hardware design, software, and more results are available at: https://sites.google.com/view/fsglove.

Problem

• Existing hand tracking systems lack sufficient degrees of freedom for complex manipulations
• Current systems fail to capture personalized hand shapes critical for contact tasks
• Commercial gloves provide insufficient joint kinematics and neglect shape variations

Innovation

• Inertial-based system tracking 48 DoFs
• DiffHCal calibration integrates with MANO model
• Differentiable optimization resolves kinematics and shape