ViSTAR: Virtual Skill Training with Augmented Reality with 3D Avatars and LLM coaching agent

2026-02-25

AI Summary

This study addresses the lack of effective feedback on movement balance, posture, and timing in autonomous basketball training by proposing an augmented reality (AR) system integrated with a Behavioral Skills Training (BST) framework. The system combines 3D motion reconstruction, multimodal AR feedback—including visual overlays, rhythmic cues, and voice guidance—and a large language model (LLM) to map spatiotemporal joint kinematics into natural-language coaching prompts. For the first time, this approach generates personalized, interpretable, real-time suggestions for performance improvement. In two user studies involving 16 participants, AI-generated feedback significantly outperformed baseline methods, with users consistently reporting that the system effectively helped them identify and correct postural and balance issues, thereby enhancing motor learning efficiency.

Technology Category

• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior
• Intelligent Robots: Behavior Learning & Control

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

We present ViSTAR, a Virtual Skill Training system in AR that supports self-guided basketball skill practice, with feedback on balance, posture, and timing. From a formative study with basketball players and coaches, the system addresses three challenges: understanding skills, identifying errors, and correcting mistakes. ViSTAR follows the Behavioral Skills Training (BST) framework-instruction, modeling, rehearsal, and feedback. It provides feedback through visual overlays, rhythm and timing cues, and an AI-powered coaching agent using 3D motion reconstruction. We generate verbal feedback by analyzing spatio-temporal joint data and mapping features to natural-language coaching cues via a Large Language Model (LLM). A key novelty is this feedback generation: motion features become concise coaching insights. In two studies (N=16), participants generally preferred our AI-generated feedback to coach feedback and reported that ViSTAR helped them notice posture and balance issues and refine movements beyond self-observation.

Problem

• skill training
• error identification
• motion feedback
• posture correction
• self-guided learning

Innovation

• Augmented Reality
• Large Language Model
• 3D Motion Reconstruction
• Behavioral Skills Training
• AI Coaching Agent