DexForce: Extracting Force-informed Actions from Kinesthetic Demonstrations for Dexterous Manipulation

2025-01-17

AI Summary

Existing imitation learning approaches suffer from poor demonstration quality in dexterous manipulation tasks characterized by rich contact interactions and fine-grained force control, primarily because kinesthetic teaching lacks authentic tactile feedback. To address this, we propose a force-aware kinesthetic teaching framework that explicitly incorporates contact forces into the motion decoding process—establishing a joint force-motion representation. We further design a lightweight, hardware-agnostic method for force information extraction, leveraging dynamics modeling to map measured forces to corresponding actions without requiring physical sensor modifications. Evaluated on six high-precision tasks—including opening an AirPods case and tightening a nut—policies trained with force-augmented demonstrations achieve an average success rate of 76%, substantially outperforming force-agnostic baselines (≈0%). This demonstrates the critical role of force observables in enhancing dexterous manipulation performance.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Humans and AI: Interaction Techniques and Devices

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Imitation learning requires high-quality demonstrations consisting of sequences of state-action pairs. For contact-rich dexterous manipulation tasks that require fine-grained dexterity, the actions in these state-action pairs must produce the right forces. Current widely-used methods for collecting dexterous manipulation demonstrations are difficult to use for demonstrating contact-rich tasks due to unintuitive human-to-robot motion retargeting and the lack of direct haptic feedback. Motivated by this, we propose DexForce, a method for collecting demonstrations of contact-rich dexterous manipulation. DexForce leverages contact forces, measured during kinesthetic demonstrations, to compute force-informed actions for policy learning. We use DexForce to collect demonstrations for six tasks and show that policies trained on our force-informed actions achieve an average success rate of 76% across all tasks. In contrast, policies trained directly on actions that do not account for contact forces have near-zero success rates. We also conduct a study ablating the inclusion of force data in policy observations. We find that while using force data never hurts policy performance, it helps the most for tasks that require an advanced level of precision and coordination, like opening an AirPods case and unscrewing a nut.

Problem

• Imitation Learning
• Tactile Sensing
• Robot Training

Innovation

• DexForce
• Contact-rich Manipulation
• Force-aware Training