Towards Human-level Dexterous Teleoperation

2026-07-13

AI Summary

Existing teleoperation systems struggle to achieve human-level dexterous in-hand manipulation, particularly in complex contact-rich tasks such as tool use and object reorientation. This work proposes TeleDexter, a hand–object co-tracking controller that maps operator intent to learned low-level contact policies. By integrating hybrid sparse-dense rewards, stochastic action masking, and domain randomization, TeleDexter enables zero-shot transfer to real robots within a single-stage reinforcement learning framework. The approach supports long-horizon teleoperation with high success rates and can be directly leveraged for behavior cloning to train autonomous policies. Evaluated on seven challenging dexterous manipulation tasks, TeleDexter achieves an average success rate of 75%, substantially outperforming current baselines.

Abstract

Humans routinely wield tools, swap grasps, and reposition objects within a single hand, seamlessly orchestrating contact transitions that span translation, reorientation, and finger gaiting. Endowing robot dexterous hands with this level of in-hand dexterity through teleoperation requires precise control of object motion via dynamic hand-object contact, yet current teleoperation systems remain far from this capability. To bridge this gap, we take a major step towards human-level dexterous teleoperation by introducing TeleDexter, a hand-object co-tracking controller that maps operator intent into learned, low-level contact execution. The controller is trained on consecutive co-tracking subgoals derived from human reference motions, utilizing a hybrid reward that couples sparse subgoal objectives with dense tracking rewards to enable learning across diverse interaction modalities rather than frame-wise trajectory imitation. The entire pipeline requires only single-stage RL and, with random action masking and domain randomization, transfers zero-shot to the real robot. We evaluate TeleDexter on seven challenging dexterous teleoperation tasks spanning object reorientation and long-horizon tool use across two dexterous hands, achieving a 75% average success rate where all baselines consistently fail. Furthermore, the collected demonstrations successfully train autonomous policies via behavioral cloning, marking a concrete step towards human-level dexterous teleoperation.

Problem

• dexterous teleoperation
• in-hand manipulation
• hand-object contact
• human-level dexterity
• robotic teleoperation

Innovation

• dexterous teleoperation
• hand-object co-tracking
• hybrid reward
• zero-shot transfer
• behavioral cloning