GazeGrasp: DNN-Driven Robotic Grasping with Wearable Eye-Gaze Interface

2025-01-13

AI Summary

To address the challenge faced by users with motor impairments in manually operating robotic systems for object grasping, this paper proposes a hands-free gaze-to-grasp system. The system leverages a wearable eye-tracking interface and introduces a novel magnet-like target alignment mechanism to significantly reduce gaze localization latency. It further incorporates personalized calibration and a closed-loop control strategy driven by gaze-and-hand gestures to enhance operational robustness and naturalness. Technically, the system integrates ESP32-CAM–based hardware eye tracking, MediaPipe for real-time gaze point estimation, YOLOv8 for object detection, and UR10 robotic arm control. In a user study with 13 participants, the magnet-like alignment mechanism reduced target acquisition time, improving overall task efficiency by 31%. The system achieves millisecond-level response latency, lightweight deployment, and strong environmental adaptability—establishing a practical, deployable paradigm for assistive robotic interaction.

Technology Category

• Intelligent Robots: Manipulation
• Computer Vision: Motion & Tracking
• Humans and AI: Interaction Techniques and Devices

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems

Abstract

We present GazeGrasp, a gaze-based manipulation system enabling individuals with motor impairments to control collaborative robots using eye-gaze. The system employs an ESP32 CAM for eye tracking, MediaPipe for gaze detection, and YOLOv8 for object localization, integrated with a Universal Robot UR10 for manipulation tasks. After user-specific calibration, the system allows intuitive object selection with a magnetic snapping effect and robot control via eye gestures. Experimental evaluation involving 13 participants demonstrated that the magnetic snapping effect significantly reduced gaze alignment time, improving task efficiency by 31%. GazeGrasp provides a robust, hands-free interface for assistive robotics, enhancing accessibility and autonomy for users.

Problem

• Assistive Robotics
• Non-manual Control
• Object Manipulation

Innovation

• GazeGrasp
• Magnetic Attraction Effect
• Accessibility Improvement