An Augmented Reality Brain-Robot Interface for Generalist Robot Arm Manipulation

2026-06-15

AI Summary

This study addresses the limited generalizability of existing augmented reality (AR) brain–computer interface (BCI) systems, which are often confined to specific tasks and struggle to support versatile robotic manipulation in real-world settings. The authors propose a novel AR-based shared autonomy framework that integrates eye tracking with motor imagery electroencephalography (EEG) signals: users select targets via gaze, while the system leverages contextual visual cues—such as “place” and “use”—to guide multi-step everyday tasks. This work represents the first integration of AR, eye tracking, and EEG-based BCI for general-purpose robotic arm control, overcoming task-specific constraints to enable intuitive, sequential interaction. In experiments with 18 participants, the system successfully facilitated complex activities including drinking, opening drawers, and operating an oven, achieving strong usability (System Usability Scale > 70) and demonstrating the feasibility and promise of this paradigm for BCI-driven robotic assistance.

Abstract

The integration of augmented reality (AR) and EEG-based brain-computer interfaces (BCIs) offers a promising path for enabling intuitive control of robots for assistive purposes. However, existing AR brain-robot interface (BRI) systems are often constrained to task-specific structures, limiting their utility in real-world environments. We present an AR BRI designed for generalist robot arm manipulation that combines gaze-based object selection with motor imagery action control. Our system uses eye-tracking for intuitive object targeting and context-aware visual overlays ("Place" and "Use") to guide the user through tasks within a shared autonomy framework. We evaluated the interface through a feasibility study with 18 healthy participants performing three multi-step activities of daily living: drinking, using a drawer, and operating an oven. Our results demonstrate that this interaction paradigm enables effective sequential task execution and high user engagement, achieving a "Good" usability rating (SUS > 70). These findings support the feasibility of the proposed interaction paradigm for complex BCI-driven robotic assistance, and motivate future evaluation with the intended target population. Project website: https://ar-bri-manip.github.io/.

Problem

• augmented reality
• brain-robot interface
• generalist manipulation
• task-specific limitation
• robotic assistance

Innovation

• Augmented Reality
• Brain-Robot Interface
• Motor Imagery
• Gaze-based Selection
• Shared Autonomy