Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini

2026-03-25

AI Summary

This work addresses the high barrier to entry in intelligent extended reality (XR) prototyping, which stems from the complexity of integrating low-level sensors with sophisticated engines. To lower this barrier, the paper introduces XR Blocks—a modular WebXR framework that pioneers the adoption of an “ambient coding” paradigm in XR. By leveraging high-level, human-centered spatial computing primitives, XR Blocks significantly simplifies authoring. The framework integrates large language models (specifically Gemini) with multimodal interaction techniques to enable end-to-end generation of functional WebXR applications directly from natural language instructions. Experiments on the VCXR60 dataset demonstrate that users can generate interactive XR prototypes—featuring realistic mixed-reality rendering and multimodal interaction—in under one minute, confirming the approach’s effectiveness and efficiency.

Technology Category

• Intelligent Robots: Embodied AI
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Responsible Web: Consent frameworks and practices on the web

Abstract

While large language models have accelerated software development through "vibe coding", prototyping intelligent Extended Reality (XR) experiences remains inaccessible due to the friction of complex game engines and low-level sensor integration. To bridge this gap, we contribute XR Blocks, an open-source, modular WebXR framework that abstracts spatial computing complexities into high-level, human-centered primitives. Building upon this foundation, we present Vibe Coding XR, an end-to-end rapid prototyping workflow that leverages LLMs to translate natural language intent directly into functional XR software. Using a web-based interface, creators can transform high-level prompts (e.g., "create a dandelion that reacts to hand") into interactive WebXR applications in under a minute. We provide a preliminary technical evaluation on a pilot dataset (VCXR60) alongside diverse application scenarios highlighting mixed-reality realism, multi-modal interaction, and generative AI integrations. By democratizing spatial software creation, this work empowers practitioners to bypass low-level hurdles and rapidly move from "idea to reality." Code and live demos are available at https://xrblocks.github.io/gem and https://github.com/google/xrblocks.

Problem

• Extended Reality (XR)
• prototyping
• large language models
• spatial computing
• human-computer interaction

Innovation

• Vibe Coding
• XR Blocks
• WebXR
• Large Language Models
• Rapid Prototyping