Designing VR Simulation System for Clinical Communication Training with LLMs-Based Embodied Conversational Agents

2025-03-03

AI Summary

Current VR-based clinical communication training systems suffer from rigid, non-customizable content. To address this, we propose a customizable VR simulation framework integrating large language models (LLMs) with embodied conversational agents (ECAs). Grounded in user-centered research, we identified three core requirements—authentic clinical scenarios, intuitive interaction, and unpredictable dialogue—and designed the Virtual AI Patient Simulator (VAPS), enabling instructors to construct personalized training scenarios without coding. Our framework overcomes traditional VR content-generation bottlenecks, significantly enhancing real-world scenario adaptability and pedagogical scalability beyond controlled lab settings. Empirical evaluation confirms that VAPS delivers highly immersive, realistic clinician–patient dialogues, effectively supporting diverse curricular objectives. (138 words)

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Multiagent Systems: Agent Communication

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Vertical and domain-specific search

Abstract

VR simulation in Health Professions (HP) education demonstrates huge potential, but fixed learning content with little customization limits its application beyond lab environments. To address these limitations in the context of VR for patient communication training, we conducted a user-centered study involving semi-structured interviews with advanced HP students to understand their challenges in clinical communication training and perceptions of VR-based solutions. From this, we derived design insights emphasizing the importance of realistic scenarios, simple interactions, and unpredictable dialogues. Building on these insights, we developed the Virtual AI Patient Simulator (VAPS), a novel VR system powered by Large Language Models (LLMs) and Embodied Conversational Agents (ECAs), supporting dynamic and customizable patient interactions for immersive learning. We also provided an example of how clinical professors could use user-friendly design forms to create personalized scenarios that align with course objectives in VAPS and discuss future implications of integrating AI-driven technologies into VR education.

Problem

• Limited customization in VR for clinical communication training.
• Need for realistic, dynamic patient interaction in VR simulations.
• Integration of AI-driven technologies to enhance VR education.

Innovation

• VR system with LLMs for dynamic patient interactions
• Embodied Conversational Agents enhance realistic communication training
• User-friendly design forms for personalized VR scenarios