KineDex: Learning Tactile-Informed Visuomotor Policies via Kinesthetic Teaching for Dexterous Manipulation

2025-05-04

AI Summary

Addressing the challenges of acquiring fine-grained tactile data, severe visual occlusion, and the decoupling of force control from policy learning in contact-rich tasks, this paper proposes KineDex—a novel kinesthetic teaching paradigm with hand-over-hand demonstration. We introduce the first tactile-embedded kinesthetic teaching framework, integrate GAN-driven visual inpainting to mitigate hand occlusion, and enable end-to-end co-optimization of multimodal (tactile–visual) input modeling and closed-loop force control. The learned tactile-augmented visuomotor policy achieves a 74.4% average success rate on highly dexterous tasks such as toothpaste extrusion—improving upon a force-controlled baseline by 57.7%. Moreover, data collection efficiency doubles that of teleoperation, while demonstration success approaches 100%.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Multi-modal Vision
• Humans and AI: Interaction Techniques and Devices

Application Category

• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Collecting demonstrations enriched with fine-grained tactile information is critical for dexterous manipulation, particularly in contact-rich tasks that require precise force control and physical interaction. While prior works primarily focus on teleoperation or video-based retargeting, they often suffer from kinematic mismatches and the absence of real-time tactile feedback, hindering the acquisition of high-fidelity tactile data. To mitigate this issue, we propose KineDex, a hand-over-hand kinesthetic teaching paradigm in which the operator's motion is directly transferred to the dexterous hand, enabling the collection of physically grounded demonstrations enriched with accurate tactile feedback. To resolve occlusions from human hand, we apply inpainting technique to preprocess the visual observations. Based on these demonstrations, we then train a visuomotor policy using tactile-augmented inputs and implement force control during deployment for precise contact-rich manipulation. We evaluate KineDex on a suite of challenging contact-rich manipulation tasks, including particularly difficult scenarios such as squeezing toothpaste onto a toothbrush, which require precise multi-finger coordination and stable force regulation. Across these tasks, KineDex achieves an average success rate of 74.4%, representing a 57.7% improvement over the variant without force control. Comparative experiments with teleoperation and user studies further validate the advantages of KineDex in data collection efficiency and operability. Specifically, KineDex collects data over twice as fast as teleoperation across two tasks of varying difficulty, while maintaining a near-100% success rate, compared to under 50% for teleoperation.

Problem

• Collecting tactile-rich demos for dexterous contact-rich manipulation
• Overcoming kinematic mismatches in prior teleoperation methods
• Enabling precise force control via kinesthetic teaching

Innovation

• Hand-over-hand kinesthetic teaching for tactile feedback
• Inpainting technique to resolve visual occlusions
• Tactile-augmented visuomotor policy for force control