ImaginateAR: AI-Assisted In-Situ Authoring in Augmented Reality

2025-04-30

AI Summary

To address the challenge of enabling non-expert users to conveniently author personalized content in augmented reality (AR), this paper proposes an on-device lightweight AI-augmented co-creation paradigm. The method integrates offline scene understanding, neural radiance field (NeRF)-accelerated real-time 3D mesh generation, and fine-tuned multimodal large language models (LLMs) for speech-driven, context-aware interaction. It eliminates reliance on fixed asset libraries or manual 3D modeling, allowing users to intuitively perceive environments and generate, refine, and iterate 3D content in arbitrary real-world scenes via natural speech. Experiments demonstrate a 27% improvement in outdoor scene graph recognition accuracy and 3.1× faster 3D generation compared to state-of-the-art methods. A user study with 20 participants confirms significant advantages in supporting open-ended creative preferences and effective AI-human co-design.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

While augmented reality (AR) enables new ways to play, tell stories, and explore ideas rooted in the physical world, authoring personalized AR content remains difficult for non-experts, often requiring professional tools and time. Prior systems have explored AI-driven XR design but typically rely on manually-defined environments and fixed asset libraries, limiting creative flexibility and real-world relevance. We introduce ImaginateAR, a mobile AI-assisted AR authoring system that aims to let anyone build anything, anywhere -- simply by speaking their imagination. ImaginateAR is powered by custom pipelines for offline scene understanding, fast 3D asset generation, and LLM-driven speech interaction. Users might say"a dragon enjoying a campfire"(P7) and iteratively refine the scene using both AI and manual tools. Our technical evaluation shows that ImaginateAR produces more accurate outdoor scene graphs and generates 3D meshes faster than prior methods. A three-part user study (N=20) revealed preferred roles for AI in authoring, what and how users create in free-form use, and design implications for future AR authoring tools.

Problem

• Simplifying AR content creation for non-experts
• Overcoming limitations of fixed asset libraries
• Enabling real-time, speech-driven AR authoring

Innovation

• Mobile AI-assisted AR authoring system
• Offline scene understanding pipelines
• LLM-driven speech interaction