ExoStart: Efficient learning for dexterous manipulation with sensorized exoskeleton demonstrations

2025-06-13

AI Summary

Dexterous manipulation by high-degree-of-freedom robotic hands in complex, multi-contact environments remains challenging. Method: This paper proposes a human-demonstration-driven zero-shot transfer learning framework comprising: (1) a novel robot-in-the-loop-free sensorized exoskeleton paradigm for efficient, high-fidelity human hand motion capture; (2) physics-based simulation dynamics filtering to convert raw demonstrations into dynamically feasible trajectories; and (3) a sparse-reward-driven self-paced reinforcement learning algorithm (a PPO variant) trained in domain-randomized simulation. Contribution/Results: The learned policy transfers zero-shot to physical hardware. Experiments on multi-contact tasks—including opening an AirPods case and inserting/rotating a key in a lock—achieve >50% success rates on real robots, substantially outperforming baselines. The approach demonstrates strong generalization across unseen objects and contact configurations, as well as high deployment efficiency.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Search and Optimization: Learning to Search

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Recent advancements in teleoperation systems have enabled high-quality data collection for robotic manipulators, showing impressive results in learning manipulation at scale. This progress suggests that extending these capabilities to robotic hands could unlock an even broader range of manipulation skills, especially if we could achieve the same level of dexterity that human hands exhibit. However, teleoperating robotic hands is far from a solved problem, as it presents a significant challenge due to the high degrees of freedom of robotic hands and the complex dynamics occurring during contact-rich settings. In this work, we present ExoStart, a general and scalable learning framework that leverages human dexterity to improve robotic hand control. In particular, we obtain high-quality data by collecting direct demonstrations without a robot in the loop using a sensorized low-cost wearable exoskeleton, capturing the rich behaviors that humans can demonstrate with their own hands. We also propose a simulation-based dynamics filter that generates dynamically feasible trajectories from the collected demonstrations and use the generated trajectories to bootstrap an auto-curriculum reinforcement learning method that relies only on simple sparse rewards. The ExoStart pipeline is generalizable and yields robust policies that transfer zero-shot to the real robot. Our results demonstrate that ExoStart can generate dexterous real-world hand skills, achieving a success rate above 50% on a wide range of complex tasks such as opening an AirPods case or inserting and turning a key in a lock. More details and videos can be found in https://sites.google.com/view/exostart.

Problem

• Teleoperating robotic hands remains challenging due to high degrees of freedom.
• Leveraging human dexterity to improve robotic hand control via exoskeleton demonstrations.
• Generating dynamically feasible trajectories for complex manipulation tasks.

Innovation

• Uses sensorized exoskeleton for direct human demonstrations
• Simulation filter creates feasible robot trajectories
• Auto-curriculum RL with sparse rewards trains policies