Audience Amplified: Virtual Audiences in Asynchronously Performed AR Theater

2024-10-21

International Symposium on Mixed and Augmented Reality

AI Summary

In mobile augmented reality (AR) settings, single users often lack authentic audience feedback and struggle to achieve social presence. Method: This work proposes an immersive theater experience framework leveraging virtual audiences in mobile AR. It constructs a phygital co-spatial environment enabling real-time interaction between users, AI-driven virtual dancers, and virtual audiences generated via imitation learning. Notably, it pioneers the application of imitation learning to virtual audience modeling—capturing behavioral diversity and imperfection of real crowds while synchronizing audio and motion. Contribution/Results: Experiments demonstrate that virtual audiences significantly enhance users’ perceived sociability and engagement. Embodied virtual avatars are preferred over audio-only feedback. Intriguingly, audience presence exhibits a dual effect: in certain contexts, its absence elicits stronger emotional responses. This study establishes a novel design paradigm and provides empirical evidence for socially immersive asynchronous AR performances.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior

Application Category

• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Audience reactions can considerably enhance live experiences; conversely, in anytime/anywhere augmented reality (AR) experiences, large crowds of people might not always be available to congregate. To get closer to simulating live events with large audiences, we created a mobile AR experience where users can wander around naturally and engage in AR theater with virtual audiences trained from real audiences using imitation learning. This allows us to carefully capture the essence of human imperfections and behavior in artificial intelligence (AI) audiences. The result is a novel mobile AR experience in which solitary AR users experience an augmented performance in a physical space with a virtual audience. Virtual dancers emerge from the surroundings, accompanied by a digitally simulated audience, to provide a community experience akin to immersive theater. In a pilot study, simulated human avatars were vastly preferred over just audience audio commentary. We subsequently engaged 20 participants as attendees of an AR dance performance, comparing a no-audience condition with a simulated audience of six onlookers. Through questionnaires and experience reports, we investigated user reactions and behavior. Our results demonstrate that the presence of virtual audience members caused attendees to perceive the performance as a social experience with increased interest and involvement in the event. On the other hand, for some attendees, the dance performances without the virtual audience evoked a stronger positive sentiment.

Problem

• Simulating live audience experiences in mobile AR theater performances
• Creating virtual audiences trained from real audiences using imitation learning
• Investigating user reactions to virtual audiences versus no-audience conditions

Innovation

• Mobile AR experience with virtual audiences trained by imitation learning
• AI audiences capture human imperfections and behavior
• Simulated audience enhances social experience in AR theater