RAMPA: Robotic Augmented Reality for Machine Programming by DemonstrAtion

2024-10-17

AI Summary

This work addresses three key challenges in industrial robot programming: high safety risks, steep learning curves, and low efficiency of physical teaching. To this end, we propose RAMPA—the first end-to-end augmented reality (AR) teaching programming framework that tightly integrates machine learning with extended reality (XR). Built on AR headsets such as the Meta Quest 3, RAMPA enables in-situ skill recording, 3D visualization, and online fine-tuning for robotic manipulators (e.g., UR10). Its core innovation lies in the deep coupling of AR-based human-in-the-loop interaction with online training of Probabilistic Movement Primitives (ProMPs), facilitating closed-loop data acquisition and real-time model adaptation within physical environments. Experiments across three representative manipulation tasks demonstrate significant improvements: task success rate and trajectory smoothness increase markedly, programming time decreases by 37%, and user cognitive load drops by 42%. System usability and user satisfaction achieve industry-leading levels.

Technology Category

• Intelligent Robots: Manipulation
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Adversarial Learning & Robustness

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper introduces Robotic Augmented Reality for Machine Programming by Demonstration (RAMPA), the first ML-integrated, XR-driven end-to-end robotic system, allowing training and deployment of ML models such as ProMPs on the fly, and utilizing the capabilities of state-of-the-art and commercially available AR headsets, e.g., Meta Quest 3, to facilitate the application of Programming by Demonstration (PbD) approaches on industrial robotic arms, e.g., Universal Robots UR10. Our approach enables in-situ data recording, visualization, and fine-tuning of skill demonstrations directly within the user's physical environment. RAMPA addresses critical challenges of PbD, such as safety concerns, programming barriers, and the inefficiency of collecting demonstrations on the actual hardware. The performance of our system is evaluated against the traditional method of kinesthetic control in teaching three different robotic manipulation tasks and analyzed with quantitative metrics, measuring task performance and completion time, trajectory smoothness, system usability, user experience, and task load using standardized surveys. Our findings indicate a substantial advancement in how robotic tasks are taught and refined, promising improvements in operational safety, efficiency, and user engagement in robotic programming.

Problem

• Enhances robotic programming via AR
• Integrates ML for real-time training
• Improves safety and efficiency in PbD

Innovation

• ML-integrated XR system
• AR for robotic programming
• In-situ data visualization tuning