KineFuse: Kinematic-Aware Haptic Fusion for In-Hand Occluded-Object Pose Tracking

2026-07-16

AI Summary

This work addresses the challenge of maintaining continuous 6D object pose tracking during dexterous manipulation, where visual occlusions caused by fingers often disrupt perception. To overcome this limitation, the authors propose a multimodal approach that fuses proprioception, proximal force/torque, and contact signals with vision. A structured finger-level encoder is introduced to automatically learn a cross-modal gating mechanism, enabling unsupervised disentanglement of translation (vision-dominated) and rotation (tactile-dominated) estimation. By representing tactile inputs as finger-level tokens and employing attention mechanisms, the method effectively integrates sparse tactile cues with visual observations for sequential pose tracking. Experiments demonstrate a 15-fold improvement in pose tracking accuracy under occlusion compared to baseline methods, along with significantly higher success rates in downstream object reorientation tasks, validated on a real robotic system.

Abstract

Dexterous in-hand manipulation requires continuous 6D pose tracking, yet the manipulating fingers inevitably occlude the object from the camera. We study how to structure the sparse haptic signals already available on multi-fingered hands, including proprioception, proximal force/torque, and binary contact, to complement a pretrained visual pose tracker under occlusion. We propose a kinematic-aware finger-level encoder and systematically compare it against four alternative designs through three levels of evaluation: per-frame refinement, sequential open-loop tracking, and closed-loop manipulation. Our experiments reveal that (i) per-frame evaluation cannot distinguish encoder quality, while sequential tracking amplifies architectural differences by up to 15 times; (ii) the structured encoder learns task-specific cross-modal gating, using vision exclusively for translation and dedicating one attention head to haptics for rotation, without explicit supervision; and (iii) compact finger-level tokenization with 4 tokens outperforms both flat fusion and joint-level representations, which suppress vision through norm dominance. We validate that improved tracking yields higher success in a downstream reorientation task and provide qualitative real-world demonstrations. Our project page is available at https://cold-young.github.io/kine-fuse/.

Problem

• in-hand manipulation
• occlusion
• 6D pose tracking
• haptic fusion
• visual-haptic integration

Innovation

• kinematic-aware fusion
• in-hand pose tracking
• haptic-visual integration
• finger-level tokenization
• cross-modal attention