XARP Tools: An Extended Reality Platform for Humans and AI Agents

2025-08-06

AI Summary

This paper addresses key challenges in XR development—inefficient human-developer–AI-agent collaboration, poor cross-device compatibility, and high interaction latency—by proposing XARP, the first unified XR development framework supporting human developers, AI agents, and Model Context Protocol (MCP) integration. Methodologically, XARP adopts a Python-based backend service coupled with multi-platform XR clients, leveraging WebSocket for lightweight JSON message transmission and abstracting low-level hardware details behind a high-level reactive API. Its core contributions are threefold: (1) native dual-role support for both human developers and AI agents; (2) cross-device, low-latency interaction (measured <50 ms end-to-end); and (3) plug-and-play ecosystem interoperability. The framework is open-sourced, including the XARP toolkit and comprehensive examples, and empirically validated across diverse AI systems (e.g., Llama, Claude) and real-world XR scenarios, demonstrating robust compatibility and effective human–AI co-development.

Technology Category

• Humans and AI: Explainable AI (XAI) for Human Understanding
• Multiagent Systems: Adversarial Agents
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health

Abstract

This technical report presents XARP Tools, an extended reality (XR) framework designed for human and AI developers alike. XARP comprises a server-side Python library and platform-specific XR clients. The library offers high-level APIs and communicates with clients via a JSON-based protocol over WebSockets. XR clients encapsulate device and runtime specifics, providing responsive, low-latency user interaction. XARP can be utilized in three ways: (i) as a library that abstracts XR development for humans; (ii) as a set of callable tools that allow AI agents to drive on-the-fly interactions with users; and (iii) as a Model Context Protocol server that plugs XR devices into AI ecosystems. XARP code and working examples are released openly at https://github.com/HAL-UCSB/xarp.

Problem

• Develops an XR framework for humans and AI developers
• Provides high-level APIs for XR development abstraction
• Enables AI agents to interact dynamically with XR users

Innovation

• Server-side Python library for XR development
• JSON-based WebSocket communication protocol
• XR clients for low-latency user interaction