Dexterous Teleoperation of 20-DoF ByteDexter Hand via Human Motion Retargeting

2025-07-03

AI Summary

This work addresses the challenge of dexterous teleoperation for high-degree-of-freedom (20-DoF) anthropomorphic robotic hands. We propose an optimization-driven human motion retargeting method that enables high-fidelity, real-time mapping from human upper-limb motions to the ByteDexter dexterous hand. Our approach integrates bio-inspired linkage-based actuation design with a lightweight optimization algorithm, achieving superior gesture mapping accuracy and increased control bandwidth while preserving motion naturalness and inter-joint coordination. The system supports high-quality demonstration data collection for extended, complex tasks—e.g., organizing cluttered makeup tables—yielding multimodal, high-fidelity human manipulation priors for imitation learning. Experimental evaluation demonstrates significant improvements in dexterity, real-time performance (<50 ms end-to-end latency), and task generalization across unseen manipulation scenarios. This work establishes a scalable technical paradigm for practical teleoperation of high-dimensional dexterous hands.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Interaction Techniques and Devices
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Replicating human--level dexterity remains a fundamental robotics challenge, requiring integrated solutions from mechatronic design to the control of high degree--of--freedom (DoF) robotic hands. While imitation learning shows promise in transferring human dexterity to robots, the efficacy of trained policies relies on the quality of human demonstration data. We bridge this gap with a hand--arm teleoperation system featuring: (1) a 20--DoF linkage--driven anthropomorphic robotic hand for biomimetic dexterity, and (2) an optimization--based motion retargeting for real--time, high--fidelity reproduction of intricate human hand motions and seamless hand--arm coordination. We validate the system via extensive empirical evaluations, including dexterous in-hand manipulation tasks and a long--horizon task requiring the organization of a cluttered makeup table randomly populated with nine objects. Experimental results demonstrate its intuitive teleoperation interface with real--time control and the ability to generate high--quality demonstration data. Please refer to the accompanying video for further details.

Problem

• Teleoperation system for high-DoF robotic hand control
• Real-time human motion retargeting for dexterous manipulation
• Generating high-quality human demonstrations for imitation learning

Innovation

• 20-DoF linkage-driven anthropomorphic robotic hand
• Optimization-based real-time motion retargeting
• Seamless hand-arm coordination teleoperation system