Evaluation of a Sign Language Avatar on Comprehensibility, User Experience & Acceptability

2025-08-07

AI Summary

This study investigates whether adjustable parameters enhance sign language virtual avatars’ intelligibility, user experience, and acceptability. Method: A comparative evaluation was conducted with German Sign Language users on Microsoft HoloLens 2, contrasting adjustable versus non-adjustable avatar systems using multimodal measures: physiological stress indicators (e.g., electrodermal activity), subjective rating scales, and in-depth interviews. Results: Adjustability conferred no significant gains in intelligibility or user experience; instead, it increased cognitive load and physiological stress. System utility scores were significantly lower than affective appeal ratings, and acceptance hinged primarily on baseline animation quality—especially facial expressions and mouth movements—and system usability. The study’s core contribution is the proposal of the “default intelligibility-first” design principle, underscoring the foundational role of high-fidelity non-manual features, and advocating a deaf–hearing collaborative, participatory design paradigm.

Technology Category

• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Computer Vision: Language and Vision

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

This paper presents an investigation into the impact of adding adjustment features to an existing sign language (SL) avatar on a Microsoft Hololens 2 device. Through a detailed analysis of interactions of expert German Sign Language (DGS) users with both adjustable and non-adjustable avatars in a specific use case, this study identifies the key factors influencing the comprehensibility, the user experience (UX), and the acceptability of such a system. Despite user preference for adjustable settings, no significant improvements in UX or comprehensibility were observed, which remained at low levels, amid missing SL elements (mouthings and facial expressions) and implementation issues (indistinct hand shapes, lack of feedback and menu positioning). Hedonic quality was rated higher than pragmatic quality, indicating that users found the system more emotionally or aesthetically pleasing than functionally useful. Stress levels were higher for the adjustable avatar, reflecting lower performance, greater effort and more frustration. Additionally, concerns were raised about whether the Hololens adjustment gestures are intuitive and easy to familiarise oneself with. While acceptability of the concept of adjustability was generally positive, it was strongly dependent on usability and animation quality. This study highlights that personalisation alone is insufficient, and that SL avatars must be comprehensible by default. Key recommendations include enhancing mouthing and facial animation, improving interaction interfaces, and applying participatory design.

Problem

• Evaluating sign language avatar comprehensibility and user experience
• Assessing impact of adjustable features on Hololens 2 avatar
• Identifying key factors for sign language avatar acceptability

Innovation

• Adjustable sign language avatar on Hololens 2
• Evaluated comprehensibility and user experience
• Improved mouthing and facial animation needed