Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactions

2026-02-07

AI Summary

This work addresses the challenges of scarce expert demonstration data and poor learning scalability in dexterous arm-hand systems for manipulation tasks by proposing a two-stage framework that integrates augmented reality (AR)-based teleoperation, behavior cloning pretraining, and contrastive learning–enhanced reinforcement learning. The approach introduces an event-driven reward mechanism that improves sample efficiency while effectively preventing policy collapse. Experimental results demonstrate that the proposed method significantly outperforms baseline algorithms such as PPO and SAC in both PyBullet simulation and real-world robotic platforms, achieving higher task success rates and faster inference speeds. These findings validate the method’s effectiveness, robustness, and safety in complex manipulation scenarios.

Technology Category

• Intelligent Robots: Manipulation
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Multiagent Systems: Adversarial Agents

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

This paper focuses on the scalable robot learning for manipulation in the dexterous robot arm-hand systems, where the remote human-robot interactions via augmented reality (AR) are established to collect the expert demonstration data for improving efficiency. In such a system, we present a unified framework to address the general manipulation task problem. Specifically, the proposed method consists of two phases: i) In the first phase for pretraining, the policy is created in a behavior cloning (BC) manner, through leveraging the learning data from our AR-based remote human-robot interaction system; ii) In the second phase, a contrastive learning empowered reinforcement learning (RL) method is developed to obtain more efficient and robust policy than the BC, and thus a projection head is designed to accelerate the learning progress. An event-driven augmented reward is adopted for enhancing the safety. To validate the proposed method, both the physics simulations via PyBullet and real-world experiments are carried out. The results demonstrate that compared to the classic proximal policy optimization and soft actor-critic policies, our method not only significantly speeds up the inference, but also achieves much better performance in terms of the success rate for fulfilling the manipulation tasks. By conducting the ablation study, it is confirmed that the proposed RL with contrastive learning overcomes policy collapse. Supplementary demonstrations are available at https://cyberyyc.github.io/.

Problem

• dexterous manipulation
• scalable robot learning
• human-robot interaction
• augmented reality
• expert demonstration

Innovation

• Augmented Reality (AR)
• Dexterous Manipulation
• Contrastive Learning
• Behavior Cloning
• Reinforcement Learning