In-Hand Manipulation of Articulated Tools with Dexterous Robot Hands with Sim-to-Real Transfer

2025-09-26

AI Summary

Dexterous manipulation of articulated tools (e.g., surgical graspers, staplers) remains fragile in real-world settings due to dense contact dynamics and unmodeled joint nonlinearities—such as friction, backlash, and clearance—that undermine policy robustness. Method: We propose a sensor-driven sim-to-real transfer framework: (1) pretraining dexterous hand manipulation policies via reinforcement learning in simulation; (2) introducing a joint representation encoding both proprioceptive and target-joint states; and (3) designing a cross-attention mechanism that fuses whole-hand tactile sensing, six-axis force-torque measurements, and policy intent for perception-decision co-refinement. Contribution/Results: Our approach significantly reduces reliance on precise physical modeling, enabling zero-shot generalization to unseen articulated tools and instance-level adaptation. It achieves robust real-world deployment on physical hardware, successfully manipulating diverse articulated tools under disturbances. This work establishes a scalable, sensor-centric paradigm for dexterous embodied manipulation.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reinforcement learning (RL) and sim-to-real transfer have advanced robotic manipulation of rigid objects. Yet, policies remain brittle when applied to articulated mechanisms due to contact-rich dynamics and under-modeled joint phenomena such as friction, stiction, backlash, and clearances. We address this challenge through dexterous in-hand manipulation of articulated tools using a robotic hand with reduced articulation and kinematic redundancy relative to the human hand. Our controller augments a simulation-trained base policy with a sensor-driven refinement learned from hardware demonstrations, conditioning on proprioception and target articulation states while fusing whole-hand tactile and force feedback with the policy's internal action intent via cross-attention-based integration. This design enables online adaptation to instance-specific articulation properties, stabilizes contact interactions, regulates internal forces, and coordinates coupled-link motion under perturbations. We validate our approach across a diversity of real-world examples, including scissors, pliers, minimally invasive surgical tools, and staplers. We achieve robust transfer from simulation to hardware, improved disturbance resilience, and generalization to previously unseen articulated tools, thereby reducing reliance on precise physical modeling in contact-rich settings.

Problem

• Addressing brittle policies for articulated tool manipulation
• Enabling sim-to-real transfer for contact-rich dynamics
• Adapting to instance-specific articulation properties online

Innovation

• Simulation-trained policy with hardware-learned sensor refinement
• Cross-attention fusion of tactile feedback and action intent
• Online adaptation to tool-specific articulation properties