Behavioral Engagement in VR-Based Sign Language Learning: Visual Attention as a Predictor of Performance and Temporal Dynamics

2026-03-19

AI Summary

This study addresses the effective measurement and prediction of learner engagement in virtual reality (VR)-based sign language learning and its impact on learning outcomes. Leveraging the VR sign language learning system SONAR, it introduces fine-grained visual attention trajectories as a core predictive variable for the first time. By integrating temporal dynamics and employing Pearson correlation, binomial generalized linear model (GLM) regression, and cross-user temporal aggregation of attention trajectories, the research investigates the relationship between engagement metrics and learning performance. Findings reveal that visual attention distribution and post-replay viewing duration significantly predict learning success, jointly accounting for a substantial proportion of variance in assessment scores. Moreover, the study uncovers attention peaks aligned with information density and phase-specific engagement dynamics, offering novel mechanisms and empirical support for modeling engagement in VR-based education.

Technology Category

• Computer Vision: Language and Vision
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Multimodal Learning

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

This study analyzes behavioral engagement in SONAR, a virtual reality application designed for sign language training and validation. We focus on three automatically derived engagement indicators (Visual Attention (VA), Video Replay Frequency (VRF), and Post-Playback Viewing Time (PPVT)) and examine their relationship with learning performance. Participants completed a self-paced Training phase, followed by a Validation quiz assessing retention. We employed Pearson correlation analysis to examine the relationships between engagement indicators and quiz performance, followed by binomial Generalized Linear Model (GLM) regression to assess their joint predictive contributions. Additionally, we conducted temporal analysis by aggregating moment-to-moment VA traces across all learners to characterize engagement dynamics during the learning session. Results show that VA exhibits a strong positive correlation with quiz performance,followed by PPVT, whereas VRF shows no meaningful association. A binomial GLM confirms that VA and PPVT are significant predictors of learning success, jointly explaining a substantial proportion of performance variance. Going beyond outcome-oriented analysis, we characterize temporal engagement patterns by aggregating moment-to-moment VA traces across all learners. The temporal profile reveals distinct attention peaks aligned with informationally dense segments of both training and validation videos, as well as phase-specific engagement dynamics, including initial acclimatization, oscillatory attention cycles during learning, and pronounced attentional peaks during assessment. Together, these findings highlight the central role of sustained and strategically allocated visual attention in VR-based sign language learning and demonstrate the value of behavioral trace data for understanding and predicting learner engagement in immersive environments.

Problem

• Behavioral Engagement
• Visual Attention
• Sign Language Learning
• Virtual Reality
• Learning Performance

Innovation

• Visual Attention
• Behavioral Engagement
• Virtual Reality
• Sign Language Learning
• Temporal Dynamics