Breaking the mould of Social Mixed Reality -- State-of-the-Art and Glossary

2025-07-31

AI Summary

Current mixed reality (MR) systems inadequately replicate human embodiment and social interaction, hindering deep, trust-based collaboration between humans and virtual agents. Method: This paper proposes a systematic framework for socially intelligent MR, introducing the first interdisciplinary terminology and theoretical foundation integrating virtual character modeling, autonomous agency, responsible AI, and cognitive neuroscience. It advances a novel tripartite technical paradigm—multimodal perception, embodied cognition, and ethical embedding—to jointly optimize perception, decision-making, and interaction. Contribution/Results: The framework significantly enhances the authenticity, psychological safety, and inclusivity of social interactions in MR. It establishes foundational support for MR systems across three critical dimensions: collaborative efficacy, ethical compliance, and neurocognitive plausibility. By centering human experience and social dynamics, this work advances a human-centered paradigm shift in social MR design and deployment.

Technology Category

• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

This article explores a critical gap in Mixed Reality (MR) technology: while advances have been made, MR still struggles to authentically replicate human embodiment and socio-motor interaction. For MR to enable truly meaningful social experiences, it needs to incorporate multi-modal data streams and multi-agent interaction capabilities. To address this challenge, we present a comprehensive glossary covering key topics such as Virtual Characters and Autonomisation, Responsible AI, Ethics by Design, and the Scientific Challenges of Social MR within Neuroscience, Embodiment, and Technology. Our aim is to drive the transformative evolution of MR technologies that prioritize human-centric innovation, fostering richer digital connections. We advocate for MR systems that enhance social interaction and collaboration between humans and virtual autonomous agents, ensuring inclusivity, ethical design and psychological safety in the process.

Problem

• MR lacks authentic human embodiment and socio-motor interaction
• MR needs multi-modal data and multi-agent interaction for social experiences
• Ethical, inclusive MR systems for human-virtual agent collaboration are needed

Innovation

• Multi-modal data streams for MR
• Multi-agent interaction capabilities
• Ethical design and inclusivity