ViHaTeleop: A Low-Cost, Lightweight Visual-Haptic Teleoperation System for Dexterous Manipulation Learning

2026-08-17

AI Summary

This study addresses the challenge of acquiring high-quality contact data in low-cost teleoperation by proposing a lightweight visuo-haptic system integrating SLAM-based wrist tracking, visual gesture recognition, and LRA vibrotactile feedback. Leveraging haptic perception redirection constraints and an Isaac Sim simulation proxy, the framework enables high-fidelity contact sensing and multimodal visuo-haptic policy learning. Experimental results across six tasks demonstrate success rate improvements of 2.2%–15.6%, with a notable 17% gain in peg-in-hole manipulation, alongside significantly enhanced subjective evaluations (p<0.05). These findings effectively validate the feasibility and closed-loop efficacy of contact-rich dexterous manipulation using low-cost hardware.

Abstract

Learning from demonstration is a promising approach for dexterous manipulation, but collecting high-quality contact-critical demonstrations remains difficult with low-cost teleoperation hardware. We present ViHaTeleop, a lightweight (0.7 kg), low-cost (\$550) visual-haptic teleoperation system with SLAM-based wrist tracking, camera-based hand tracking, and finger-wise vibrotactile feedback through Linear Resonant Actuators (LRA). The system includes several design choices (LED illumination, fisheye hand camera, and tactile-aware retargeting constraints) and is deployed on Franka + LEAP Hand + 9DTact in both real and simulated environments. Under matched with/without-haptic conditions with nine participants across six contact-critical tasks, haptics improved success rates across all tasks (+2.2 to +15.6 percentage points), while completion-time effects were task-dependent. Subjective ratings showed significant gains in contact clarity and grasp confidence in both simulation and real-world settings (Wilcoxon signed-rank, $p<0.05$). We also integrate a lightweight depth-camera-based tactile proxy in Isaac Sim, enabling a full pipeline from multi-modal demonstration collection to visual-tactile policy training. Preliminary downstream validation by training visual-tactile policies from collected demonstrations shows tactile cues benefit contact-critical subtasks (peg-in-hole: +17 percentage points over vision-only).

Problem

• Dexterous Manipulation
• Learning from Demonstration
• Teleoperation
• Haptic Feedback
• Contact-Critical Tasks

Innovation

• Visual-Haptic Teleoperation
• Vibrotactile Feedback
• Dexterous Manipulation Learning
• Tactile Proxy
• Low-Cost System