OrchestrXR: A Multi-Agent System for Idea-to-Prototype XR Study Authoring

2026-07-01

AI Summary

This work addresses the fragmented transition from conceptualization to functional prototypes in extended reality (XR) research, where experimental design, scene construction, and interaction implementation often proceed in isolation, lacking a unified and efficient authoring pipeline. To bridge this gap, we propose OrchestrXR—a human-AI collaborative multi-agent workflow that introduces, for the first time, a structured schema and multi-agent coordination mechanism to enable end-to-end, stage-wise controllable generation of Unity-based XR prototypes directly from research ideas. A dedicated human-AI interface ensures intent consistency across all development stages. In a user study, 12 XR researchers successfully leveraged OrchestrXR to rapidly produce early-stage prototypes with high fidelity to their original intents, demonstrating the system’s effectiveness and practical utility.

Abstract

Extended Reality (XR) has become an important interaction paradigm in Human-Computer Interaction (HCI). XR studies are used to investigate interaction, perception, and user behavior in immersive environments, and typically involve experimental tasks, 3D scenes, and interactive logic. However, turning an initial XR study idea into a runnable prototype remains fragmented across study design, scene construction, and interaction implementation. We present OrchestrXR, a multi-agent human-AI workflow for early-stage idea-to-prototype XR study authoring. Rather than treating XR study creation as one-shot generation, OrchestrXR supports a controllable workflow across study design, scene generation, and interaction generation through structured schemas, multi-agent orchestration, and interactive human-agent interfaces, producing a Unity-based prototype from a researcher's idea. A user study with 12 XR researchers suggests that OrchestrXR provides effective support for early-stage XR study authoring with strong intent preservation across stages.

Problem

• Extended Reality (XR)
• prototype authoring
• study design
• scene construction
• interaction implementation

Innovation

• multi-agent system
• idea-to-prototype
• structured schemas
• human-AI collaboration
• XR authoring