PlayWrite: A Multimodal System for AI Supported Narrative Co-Authoring Through Play in XR

2026-03-02

AI Summary

This work addresses the limitations of current AI writing tools, which rely heavily on textual prompts and struggle to support creative ideation rooted in spatial manipulation and improvisational interaction during story creation. To bridge this gap, the paper introduces a mixed-reality (XR) system that shifts the paradigm from text-based prompting to embodied narrative by enabling users to directly manipulate virtual characters and props as the primary mode of interaction. The system employs a multi-agent AI pipeline to interpret user actions into structured “intent frames,” which are then rendered into coherent narrative text by a large language model. A user study (N=13) demonstrates that this approach significantly enhances improvisational storytelling, with participants perceiving the AI as a collaborative partner whose unexpected outputs effectively stimulate novel ideas and help overcome creative blocks.

Technology Category

• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Cognitive Modeling & Cognitive Systems: Computational Creativity
• Intelligent Robots: Embodied AI

Application Category

• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Current AI writing tools, which rely on text prompts, poorly support the spatial and interactive nature of storytelling where ideas emerge from direct manipulation and play. We present PlayWrite, a mixed-reality system where users author stories by directly manipulating virtual characters and props. A multi-agent AI pipeline interprets these actions into Intent Frames -structured narrative beats visualized as rearrangeable story marbles on a timeline. A large language model then transforms the user's assembled sequence into a final narrative. A user study (N=13) with writers from varying domains found that PlayWrite fosters a highly improvisational and playful process. Users treated the AI as a collaborative partner, using its unexpected responses to spark new ideas and overcome creative blocks. PlayWrite demonstrates an approach for co-creative systems that move beyond text to embrace direct manipulation and play as core interaction modalities.

Problem

• AI writing tools
• spatial storytelling
• interactive narrative
• creative co-authoring
• direct manipulation

Innovation

• multimodal interaction
• mixed reality
• intent frames
• co-creative AI
• direct manipulation