Spot-On: A Mixed Reality Interface for Multi-Robot Cooperation

2025-05-28

AI Summary

This work addresses the challenge of multi-legged robots collaboratively manipulating physical infrastructure—such as drawers, doors, and switches—in semantically diverse real-world environments. We propose a mixed reality (MR)-driven framework for multi-robot semantic interaction, integrating lightweight SLAM, real-time semantic scene understanding, MR-based natural gesture recognition, and a distributed robot control protocol to enable cross-robot task orchestration via gesture–object binding. Our key contribution is the first MR-native paradigm for multi-robot semantic collaboration, supporting intuitive, low-cognitive-load human–robot co-manipulation. A user study demonstrates that 96% of task executions were rated “good” or “excellent,” validating the system’s high usability, robustness, and interaction intuitiveness in complex, unstructured real-world settings.

Technology Category

• Intelligent Robots: Multi-Robot Systems
• Multiagent Systems: Coordination and Collaboration
• Humans and AI: Interaction Techniques and Devices

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Recent progress in mixed reality (MR) and robotics is enabling increasingly sophisticated forms of human-robot collaboration. Building on these developments, we introduce a novel MR framework that allows multiple quadruped robots to operate in semantically diverse environments via a MR interface. Our system supports collaborative tasks involving drawers, swing doors, and higher-level infrastructure such as light switches. A comprehensive user study verifies both the design and usability of our app, with participants giving a"good"or"very good"rating in almost all cases. Overall, our approach provides an effective and intuitive framework for MR-based multi-robot collaboration in complex, real-world scenarios.

Problem

• Enabling multi-robot cooperation via mixed reality interface
• Supporting collaborative tasks in diverse semantic environments
• Providing intuitive MR framework for real-world robot collaboration

Innovation

• Mixed reality interface for multi-robot cooperation
• Quadruped robots operate in diverse environments
• Supports collaborative tasks with infrastructure