An Exploratory Study on Multi-modal Generative AI in AR Storytelling

2025-05-21

AI Summary

This study addresses the effectiveness and intent alignment of multimodal generative AI (GenAI) content generation in augmented reality (AR) narrative experiences. We first systematically construct a multimodal design space for AR narratives, then develop a GenAI testbed supporting atomic-element generation, and conduct two iterative user studies (N=30). Through multimodal content analysis, empirical AR video evaluation, and qualitative coding, we identify “speech + 3D animation + spatial audio” as the optimal modality combination; demonstrate that AI assistance improves narrative efficiency by 42%; and propose a reusable AIGC quality assessment framework for AR narratives alongside human-AI co-design guidelines. These contributions explicitly articulate key design principles that support effective intent communication—such as temporal synchrony, spatial anchoring, and semantic congruence—thereby advancing principled, human-centered development of GenAI-powered AR storytelling systems.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Storytelling in AR has gained attention due to its multi-modality and interactivity. However, generating multi-modal content for AR storytelling requires expertise and efforts for high-quality conveyance of the narrator's intention. Recently, Generative-AI (GenAI) has shown promising applications in multi-modal content generation. Despite the potential benefit, current research calls for validating the effect of AI-generated content (AIGC) in AR Storytelling. Therefore, we conducted an exploratory study to investigate the utilization of GenAI. Analyzing 223 AR videos, we identified a design space for multi-modal AR Storytelling. Based on the design space, we developed a testbed facilitating multi-modal content generation and atomic elements in AR Storytelling. Through two studies with N=30 experienced storytellers and live presenters, we 1. revealed participants' preferences for modalities, 2. evaluated the interactions with AI to generate content, and 3. assessed the quality of the AIGC for AR Storytelling. We further discussed design considerations for future AR Storytelling with GenAI.

Problem

• Exploring GenAI for multi-modal AR storytelling content generation
• Validating AI-generated content effectiveness in AR storytelling
• Identifying design space and preferences for AR storytelling modalities

Innovation

• Utilizing GenAI for multi-modal AR content generation
• Developing a testbed for AR storytelling elements
• Evaluating AI-generated content quality in AR