IRIS: An Immersive Robot Interaction System

2025-02-05

AI Summary

Existing XR-based robotic data collection systems exhibit poor reusability and scalability due to tight coupling with specific simulators and environments. To address this, we propose the first general-purpose immersive XR framework supporting cross-simulator interoperability (MuJoCo, IsaacSim, CoppeliaSim, Genesis), cross-benchmark compatibility (e.g., LIBERO), and hybrid simulation-to-reality scenarios. Our method introduces: (1) a unified scene specification and shared spatial anchoring mechanism enabling synchronized multi-XR-device simulation and low-latency fusion of real-world sensor data (e.g., depth-camera point clouds); and (2) a modular Unity-based architecture integrating XR interaction, heterogeneous simulation backends, and an optimized communication protocol. Experiments demonstrate significant improvements over baselines in both data acquisition efficiency and subjective user experience. The framework has been successfully deployed on Meta Quest 3 and Microsoft HoloLens 2.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

This paper introduces IRIS, an immersive Robot Interaction System leveraging Extended Reality (XR), designed for robot data collection and interaction across multiple simulators, benchmarks, and real-world scenarios. While existing XR-based data collection systems provide efficient and intuitive solutions for large-scale data collection, they are often challenging to reproduce and reuse. This limitation arises because current systems are highly tailored to simulator-specific use cases and environments. IRIS is a novel, easily extendable framework that already supports multiple simulators, benchmarks, and even headsets. Furthermore, IRIS is able to include additional information from real-world sensors, such as point clouds captured through depth cameras. A unified scene specification is generated directly from simulators or real-world sensors and transmitted to XR headsets, creating identical scenes in XR. This specification allows IRIS to support any of the objects, assets, and robots provided by the simulators. In addition, IRIS introduces shared spatial anchors and a robust communication protocol that links simulations between multiple XR headsets. This feature enables multiple XR headsets to share a synchronized scene, facilitating collaborative and multi-user data collection. IRIS can be deployed on any device that supports the Unity Framework, encompassing the vast majority of commercially available headsets. In this work, IRIS was deployed and tested on the Meta Quest 3 and the HoloLens 2. IRIS showcased its versatility across a wide range of real-world and simulated scenarios, using current popular robot simulators such as MuJoCo, IsaacSim, CoppeliaSim, and Genesis. In addition, a user study evaluates IRIS on a data collection task for the LIBERO benchmark. The study shows that IRIS significantly outperforms the baseline in both objective and subjective metrics.

Problem

• IRIS addresses reproducibility and reusability in XR-based data collection.
• IRIS integrates real-world sensor data with simulated environments.
• IRIS enables multi-user, synchronized XR scenes for collaborative tasks.

Innovation

• Leverages Extended Reality (XR)
• Supports multiple simulators and headsets
• Enables synchronized multi-user data collection