FIP: Endowing Robust Motion Capture on Daily Garment by Fusing Flex and Inertial Sensors

2025-02-20

AI Summary

To address the degradation in joint pose estimation accuracy caused by sensor displacement in loose-fitting daily apparel, this paper proposes the Flexible Inertial Pose Estimator (FIP). FIP introduces a novel displacement latent-variable diffusion model that jointly models and compensates for positional discrepancies between bend sensors and IMUs. It incorporates a physics-guided calibrator to enforce kinematic consistency constraints and designs a multimodal temporal fusion predictor that integrates bend-sensor signals, IMU measurements, and latent-variable representations. On standard benchmarks, FIP achieves state-of-the-art improvements of 19.5%, 26.4%, and 30.1% in overall angular error, elbow angular error, and end-effector position error, respectively. Moreover, FIP demonstrates strong robustness across diverse body morphologies and dynamic motion sequences.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Biometrics, Face, Gesture & Pose
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

What if our clothes could capture our body motion accurately? This paper introduces Flexible Inertial Poser (FIP), a novel motion-capturing system using daily garments with two elbow-attached flex sensors and four Inertial Measurement Units (IMUs). To address the inevitable sensor displacements in loose wearables which degrade joint tracking accuracy significantly, we identify the distinct characteristics of the flex and inertial sensor displacements and develop a Displacement Latent Diffusion Model and a Physics-informed Calibrator to compensate for sensor displacements based on such observations, resulting in a substantial improvement in motion capture accuracy. We also introduce a Pose Fusion Predictor to enhance multimodal sensor fusion. Extensive experiments demonstrate that our method achieves robust performance across varying body shapes and motions, significantly outperforming SOTA IMU approaches with a 19.5% improvement in angular error, a 26.4% improvement in elbow angular error, and a 30.1% improvement in positional error. FIP opens up opportunities for ubiquitous human-computer interactions and diverse interactive applications such as Metaverse, rehabilitation, and fitness analysis.

Problem

• Enhance motion capture accuracy
• Compensate sensor displacements in wearables
• Improve multimodal sensor fusion efficiency

Innovation

• Fuses flex and inertial sensors
• Develops Displacement Latent Diffusion Model
• Introduces Pose Fusion Predictor