Smooth Operator: A Real-Time Sampling-Based Algorithm for Kinematic Hand Retargeting

2026-07-08

AI Summary

This work addresses the limitations of existing gradient-based hand retargeting methods, which are prone to local optima and induce motion jitter, thereby degrading teleoperation data quality and user experience. To overcome these issues, the authors propose a novel gradient-free, sampling-based hand retargeting approach (SBR), which introduces sampling-based control theory into this domain for the first time. By leveraging a sampling-driven optimization strategy that circumvents explicit gradient computation, SBR achieves low-jitter, real-time kinematic mapping. Experimental results demonstrate that SBR significantly outperforms current baselines, achieving the highest task success rate of 54.1% in real-user studies and the lowest NASA-TLX cognitive workload score (36.4/100). Furthermore, this study establishes the first systematic benchmark for evaluating hand retargeting performance.

Abstract

Advances in learning-based robotic manipulation, such as Vision-Language-Action (VLA) models and Video Action Models (VAMs), heavily rely on high-quality teleoperation data. Their capabilities are strictly upper-bounded by the quality of the underlying human demonstrations. Current gradient-based retargeting algorithms often converge to different local minima, resulting in jitter that affects data quality and teleoperation experience. To address this, we introduce the Sampling-Based Retargeter (SBR), a novel gradient-free retargeting method drawn from the rich literature of sampling-based control and explicitly designed for low-jitter, real-time kinematic retargeting. We evaluate SBR both in simulation and through a rigorous real-world user study involving 18 participants performing 3 complex manipulation tasks. Compared to gradient-based baselines, SBR achieved the highest overall task success rate (54.1%) while significantly reducing operator cognitive fatigue, recording the lowest NASA-TLX workload score (36.4 out of 100). Ultimately, we establish SBR as a highly effective, intuitive retargeter for dexterous manipulation, providing the community with a rigorous benchmarking methodology to guide future retargeting research.

Problem

• kinematic retargeting
• jitter
• teleoperation
• human demonstration
• robotic manipulation

Innovation

• sampling-based retargeting
• kinematic hand retargeting
• gradient-free optimization
• teleoperation
• real-time manipulation