TeleDex: Accessible Dexterous Teleoperation

2026-03-17

AI Summary

This work proposes a low-cost, accessible teleoperation framework for dexterous robotic manipulation that leverages only a standard consumer smartphone, circumventing the need for expensive dedicated hardware. By fusing the phone’s built-in visual and inertial sensors, the system estimates full 6-degree-of-freedom (DoF) wrist pose and 21-DoF hand articulation in real time, enabling direct mapping of human gestures to a robotic arm and dexterous hand without external tracking or complex calibration. The approach supports both handheld and 3D-printed finger-mounted configurations, offering low latency, infrastructure-free operation, and fully open-source software and hardware integration. Demonstrations in both simulation and real-world settings successfully execute diverse manipulation tasks, marking the first instance of driving a high-DoF dexterous hand using only an off-the-shelf smartphone. This significantly lowers the barrier to collecting demonstration data and provides a unified interface for scalable teleoperation.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Biometrics, Face, Gesture & Pose
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Despite increasing dataset scale and model capacity, robot manipulation policies still struggle to generalize beyond their training distributions. As a result, deploying state-of-the-art policies in new environments, tasks, or robot embodiments often requires collecting additional demonstrations. Enabling this in real-world deployment settings requires tools that allow users to collect demonstrations quickly, affordably, and with minimal setup. We present TeleDex, an open-source system for intuitive teleoperation of dexterous hands and robotic manipulators using any readily available phone. The system streams low-latency 6-DoF wrist poses and articulated 21-DoF hand state estimates from the phone, which are retargeted to robot arms and multi-fingered hands without requiring external tracking infrastructure. TeleDex supports both a handheld phone-only mode and an optional 3D-printable hand-mounted interface for finger-level teleoperation. By lowering the hardware and setup barriers to dexterous teleoperation, TeleDex enables users to quickly collect demonstrations during deployment to support policy fine-tuning. We evaluate the system across simulation and real-world manipulation tasks, demonstrating its effectiveness as a unified scalable interface for robot teleoperation. All software and hardware designs, along with demonstration videos, are open-source and available at orayyan.com/teledex.

Problem

• robot manipulation
• generalization
• teleoperation
• demonstration collection
• real-world deployment

Innovation

• teleoperation
• dexterous manipulation
• smartphone-based interface
• retargeting
• open-source robotics