ERUPT: An Open Toolkit for Interfacing with Robot Motion Planners in Extended Reality

2025-10-02

AI Summary

This study addresses key challenges in robot motion planning—rigid environmental modeling, limited spatial understanding, and disconnection between simulation and physical deployment—by proposing an immersive, XR-enabled interactive planning system. Methodologically, it introduces the first deep integration of MoveIt into an XR environment, enabling users to dynamically construct and reconfigure 3D scenes via natural hand gestures and voice commands, while performing real-time motion planning, path editing, and validation. The system supports multimodal interaction, collision-free simulation-based rehearsal, and one-click deployment to physical robots. Contributions include: (1) the first native MoveIt-XR coupling framework; (2) an immersive human-robot co-planning paradigm that enables joint evolution of environment and task specifications; and (3) substantial improvements in planning intuitiveness, spatial cognition efficiency, and simulation-to-reality consistency. Experimental evaluation demonstrates effectiveness and practicality in complex indoor scenarios.

Technology Category

• Intelligent Robots: Motion and Path Planning
• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Multiagent Systems: Multiagent Planning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health

Abstract

We propose the Extended Reality Universal Planning Toolkit (ERUPT), an extended reality (XR) system for interactive motion planning. Our system allows users to create and dy- namically reconfigure environments while they plan robot paths. In immersive three-dimensional XR environments, users gain a greater spatial understanding. XR also unlocks a broader range of natural interaction capabilities, allowing users to grab and adjust objects in the environment similarly to the real world, rather than using a mouse and keyboard with the scene projected onto a two-dimensional computer screen. Our system integrates with MoveIt, a manipulation planning framework, allowing users to send motion planning requests and visualize the resulting robot paths in virtual or augmented reality. We provide a broad range of interaction modalities, allowing users to modify objects in the environment and interact with a virtual robot. Our system allows operators to visualize robot motions, ensuring desired behavior as it moves throughout the environment, without risk of collisions within a virtual space, and to then deploy planned paths on physical robots in the real world.

Problem

• Creating interactive XR environments for robot motion planning
• Enabling natural 3D interaction instead of 2D screen controls
• Visualizing and deploying collision-free robot paths in reality

Innovation

• Extended Reality toolkit for interactive robot motion planning
• Integrates with MoveIt framework for planning and visualization
• Allows virtual environment modification and collision-free path deployment