LLM-Powered Virtual Patient Agents for Interactive Clinical Skills Training with Automated Feedback

2025-08-19

AI Summary

Traditional OSCE implementation relies on costly resources—such as standardized patients and expert feedback—while existing LLM-driven text-based virtual patients fail to model authentic non-textual interactions, limiting training realism and dynamism. This paper proposes a multimodal virtual patient agent framework: (1) an agent model with an explicit action space enabling expressive non-verbal behavior modeling (e.g., facial expressions, posture); and (2) a real-time responsive virtual tutor agent delivering personalized, context-aware clinical skill feedback. Empirical evaluation demonstrates low system latency and high component accuracy. Expert medical reviewers confirm the virtual patient’s natural, coherent behavior and the tutor’s clinically actionable feedback. The framework substantially lowers barriers to autonomous OSCE training, enabling medical students to conduct efficient, low-cost, high-fidelity clinical skills practice remotely.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Multiagent Systems: Agent Communication
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Objective Structured Clinical Examinations (OSCEs) are essential for medical training, but they require significant resources, including professional actors and expert medical feedback. Although Large Language Models (LLMs) have introduced text-based virtual patients for communication practice, these simulations often lack the capability for richer, non-textual interactions. This paper presents a novel framework that significantly enhances LLM-based simulated patients by equipping them with action spaces, thereby enabling more realistic and dynamic patient behaviors that extend beyond text. Furthermore, our system incorporates virtual tutors that provide students with instant, personalized feedback on their performance at any time during these simulated encounters. We have conducted a rigorous evaluation of the framework's real-time performance, including system latency and component accuracy. Preliminary evaluations with medical experts assessed the naturalness and coherence of the simulated patients, as well as the usefulness and appropriateness of the virtual tutor's assessments. This innovative system provides medical students with a low-cost, accessible platform for personalized OSCE preparation at home.

Problem

• Enhancing LLM-based virtual patients with action spaces for realistic interactions
• Providing instant personalized feedback during simulated clinical encounters
• Reducing resource requirements for Objective Structured Clinical Examinations training

Innovation

• Enhanced LLM-based patients with action spaces
• Virtual tutors for instant personalized feedback
• Low-cost platform for home OSCE preparation