Interactive Robot Programming for Surface Finishing via Task-Centric Mixed Reality Interfaces

2025-12-29

AI Summary

In small-batch, high-variability surface finishing tasks, existing collaborative robot programming heavily relies on expert skills, hindering adoption by non-expert users. Method: This paper proposes a task-oriented mixed reality (MR) programming framework that integrates a lightweight, human-correctable surface segmentation algorithm with a task-centric MR interaction paradigm. Users perform end-to-end surface identification, receive real-time 3D visual feedback, and generate adaptive toolpaths via natural hand gestures and voice commands—eliminating dependence on traditional offline programming and manual teaching. Contribution/Results: The approach significantly reduces cognitive load. A user study demonstrates that novice users can configure high-quality finishing tasks within five minutes, achieving a 3.2× improvement in programming efficiency and attaining a System Usability Scale (SUS) score of 86.4.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Planning, Routing, and Scheduling: Mixed Discrete/Continuous Planning
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Lengthy setup processes that require robotics expertise remain a major barrier to deploying robots for tasks involving high product variability and small batch sizes. As a result, collaborative robots, despite their advanced sensing and control capabilities, are rarely used for surface finishing in small-scale craft and manufacturing settings. To address this gap, we propose a novel robot programming approach that enables non-experts to intuitively program robots through interactive, task-focused workflows. For that, we developed a new surface segmentation algorithm that incorporates human input to identify and refine workpiece regions for processing. Throughout the programming process, users receive continuous visual feedback on the robot's learned model, enabling them to iteratively refine the segmentation result. Based on the segmented surface model, a robot trajectory is generated to cover the desired processing area. We evaluated multiple interaction designs across two comprehensive user studies to derive an optimal interface that significantly reduces user workload, improves usability and enables effective task programming even for users with limited practical experience.

Problem

• Enables non-experts to program robots via interactive mixed reality interfaces
• Develops a surface segmentation algorithm incorporating human input for workpieces
• Generates robot trajectories from segmented models to cover processing areas

Innovation

• Mixed reality interface for intuitive robot programming
• Human-in-the-loop surface segmentation algorithm
• Visual feedback system for iterative task refinement