IMU-HOI: A Symbiotic Framework for Coherent Human-Object Interaction and Motion Capture via Contact-Conscious Inertial Fusion

2026-06-26

AI Summary

Existing inertial motion capture methods struggle to jointly recover human pose and object motion due to their neglect of contact and dynamic constraints inherent in human-object interactions. This work proposes a sparse-IMU-based joint estimation framework that, for the first time in a purely inertial system, incorporates a contact-aware mechanism: it infers hand-object contact probability from IMU data as a high-level cue to guide motion inference, and integrates hand forward kinematics with object IMU integration within a three-stage fusion optimization pipeline. The method enables drift-resistant, highly consistent co-reconstruction of full-body pose and six-degree-of-freedom object trajectories without any visual input, significantly outperforming existing inertial approaches in complex interaction scenarios while seamlessly integrating into current sparse-IMU setups.

Abstract

Capturing full-body human motion with object interactions is crucial for AR/VR and robotics applications, yet it remains challenging for conventional vision-based methods due to occlusions and constrained capture volumes. Inertial measurement units (IMUs) offer a compelling alternative without line-of-sight requirements, but existing IMU-based motion capture assumes an isolated human and ignores object contacts and dynamics. To bridge this gap, we present IMU-HOI, a novel framework that jointly recovers full-body human pose and 6-DoF object trajectory from sparse IMUs on the body and object, explicitly modeling human-object interaction. Our approach first infers probabilistic hand-object contacts directly from IMU streams and uses them as a high-level signal to route between kinematic and inertial reasoning. These contact cues drive a three-stage fusion pipeline that refines human pose and root translation, and fuses hand-based forward kinematics with object-IMU integration for object motion, yielding coherent, drift-resilient trajectories for both human and object. Experiments on challenging human-object interaction scenarios demonstrate substantial accuracy gains over prior inertial motion capture methods. Moreover, IMU-HOI can be plugged into existing sparse-IMU mocap backbones with minimal changes, effectively extending the scope of purely inertial motion capture from isolated humans to full human-object interaction and joint motion estimation.

Problem

• Human-Object Interaction
• Inertial Motion Capture
• IMU
• Contact Modeling
• 6-DoF Object Tracking

Innovation

• IMU-based motion capture
• human-object interaction
• contact-aware fusion
• 6-DoF object tracking
• inertial sensing