Skill-Adaptive Ghost Instructors: Enhancing Retention and Reducing Over-Reliance in VR Piano Learning

2026-03-06

AI Summary

This work addresses the overreliance on continuous guidance in existing XR-based motor skill learning systems, which often impedes skill internalization and transfer. To mitigate this issue, the authors propose a skill-adaptive, dynamically transparent “ghost tutor” that introduces, for the first time, a real-time performance–driven transparency modulation mechanism. Implemented in a VR piano fingering training task, the system intelligently balances instructional support with autonomous practice by integrating virtual reality, real-time performance assessment, and dynamic visual guidance control, delivering adaptive feedback from a first-person perspective. In a user study with 30 participants, the proposed approach significantly improved pitch and fingering accuracy, effectively curbed error accumulation, and demonstrated superior short-term retention compared to fixed-transparency baselines.

Technology Category

• Intelligent Robots: Manipulation
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Machine Learning: Transparent, Interpretable, Explainable ML

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Motor-skill learning systems in XR rely on persistent cues. However, constant cueing can induce overreliance and erode memorization and skill transfer. We introduce a skill-adaptive, dynamically transparent ghost instructor whose opacity adapts in real time to learner performance. In a first-person perspective, users observe a ghost hand executing piano fingering with either a static or a performance-adaptive transparency in a VR piano training application. We conducted a within-subjects study (N=30), where learners practiced with traditional Static (fixed-transparency) and our proposed Dynamic (performance-adaptive) modes and were tested without guidance immediately and after a 10-minute retention interval. Relative to Static, the Dynamic mode yielded higher pitch and fingering accuracy and limited error increases, with comparable timing. These findings suggest that adaptive transparency helps learners internalize fingerings more effectively, reducing dependency on external cues and improving short-term skill retention within immersive learning environments. We discuss design implications for motor-skill learning and outline directions for extending this approach to longer-term retention and more complex tasks.

Problem

• motor-skill learning
• over-reliance
• skill retention
• XR
• external cues

Innovation

• skill-adaptive
• dynamic transparency
• ghost instructor
• motor-skill learning
• virtual reality