Whispers of the Butterfly: A Research-through-Design Exploration of In-Situ Conversational AI Guidance in Large-Scale Outdoor MR Exhibitions

2026-02-05

AI Summary

This study addresses the limitations of traditional guided tours in large-scale outdoor mixed reality (MR) art exhibitions, where scripted approaches struggle to scale with exploratory complexity. To overcome this, the authors developed Dream-Butterfly—a summonable, non-human embodied conversational AI guide—through research-through-design. This multilingual agent delivers dynamic, context-aware narration grounded in exhibition content. A controlled field experiment with 24 visitors compared experiences between AI and human-guided tours during a real outdoor MR exhibition, revealing differences in information recall, perceived responsiveness, immersion, and cognitive load. The findings further illuminate how visitors negotiate guiding responsibilities among staff, AI, and themselves. The project contributes transferable design strategies for deploying embodied conversational agents under constraints of mobility and safety, offering a new paradigm for interactive cultural mediation in immersive environments.

Technology Category

• Intelligent Robots: Embodied AI
• Humans and AI: Interaction Techniques and Devices
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval

Application Category

• Search and Retrieval-Augmented AI: Assisted, interactive, and conversational search
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Large-scale outdoor mixed reality (MR) art exhibitions distribute curated virtual works across open public spaces, but interpretation rarely scales without turning exploration into a scripted tour. Through Research-through-Design, we created Dream-Butterfly, an in-situ conversational AI docent embodied as a small non-human companion that visitors summon for multilingual, exhibition-grounded explanations. We deployed Dream-Butterfly in a large-scale outdoor MR exhibition at a public university campus in southern China, and conducted an in-the-wild between-subject study (N=24) comparing a primarily human-led tour with an AI-led tour while keeping staff for safety in both conditions. Combining questionnaires and semi-structured interviews, we characterize how shifting the primary explanation channel reshapes explanation access, perceived responsiveness, immersion, and workload, and how visitors negotiate responsibility handoffs among staff, the AI guide, and themselves. We distill transferable design implications for configuring mixed human-AI guiding roles and embodying conversational agents in mobile, safety-constrained outdoor MR exhibitions.

Problem

• mixed reality
• outdoor exhibition
• conversational AI
• interpretation
• guidance

Innovation

• conversational AI
• mixed reality
• Research-through-Design
• embodied agent
• human-AI collaboration