Reframing Human-Robot Interaction Through Extended Reality: Unlocking Safer, Smarter, and More Empathic Interactions with Virtual Robots and Foundation Models

2025-12-02

AI Summary

Physical robots face inherent limitations in hardware constraints, safety assurance, empathic capability, and cross-context adaptability. Method: This work proposes a novel virtual robot paradigm centered on extended reality (XR) as the embodiment medium and large foundation models (LFMs) as the cognitive core. We integrate multimodal large models, biosensing, and contextual reasoning to construct an evolvable, embodied, and affect-aware virtual agent system—positioning the virtual robot as a “cognitive and empathic mediator” for situational understanding, dynamic affective response, and long-term adaptive interaction. Contribution/Results: Empirical evaluation demonstrates effectiveness in safety-critical and socially interactive scenarios. We further introduce a human-centered, ethics-embedded evaluation framework. The work advances the evolution of human–robot interaction toward tighter physical–virtual integration, higher adaptability, and stronger empathic intelligence.

Technology Category

• Intelligent Robots: Human-Robot Interaction
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search

Abstract

This perspective reframes human-robot interaction (HRI) through extended reality (XR), arguing that virtual robots powered by large foundation models (FMs) can serve as cognitively grounded, empathic agents. Unlike physical robots, XR-native agents are unbound by hardware constraints and can be instantiated, adapted, and scaled on demand, while still affording embodiment and co-presence. We synthesize work across XR, HRI, and cognitive AI to show how such agents can support safety-critical scenarios, socially and cognitively empathic interaction across domains, and outreaching physical capabilities with XR and AI integration. We then discuss how multimodal large FMs (e.g., large language model, large vision model, and vision-language model) enable context-aware reasoning, affect-sensitive situations, and long-term adaptation, positioning virtual robots as cognitive and empathic mediators rather than mere simulation assets. At the same time, we highlight challenges and potential risks, including overtrust, cultural and representational bias, privacy concerns around biometric sensing, and data governance and transparency. The paper concludes by outlining a research agenda for human-centered, ethically grounded XR agents - emphasizing multi-layered evaluation frameworks, multi-user ecosystems, mixed virtual-physical embodiment, and societal and ethical design practices to envision XR-based virtual agents powered by FMs as reshaping future HRI into a more efficient and adaptive paradigm.

Problem

• Enhancing human-robot interaction safety and empathy via virtual agents.
• Overcoming hardware limits with scalable, adaptive XR-native robots.
• Addressing ethical risks like bias and privacy in AI-driven XR systems.

Innovation

• Using extended reality to create virtual robots
• Leveraging foundation models for cognitive and empathic agents
• Integrating XR and AI for safe, adaptive interactions