Sentiment Analysis of German Sign Language Fairy Tales

2026-04-17

AI Summary

This study addresses the underexplored role of bodily movements in conveying emotions within German Sign Language (DGS), a domain traditionally dominated by facial expression analysis. The authors leverage large language models to annotate emotional categories in German fairy tale texts and extract facial and full-body pose keypoints from corresponding DGS videos using MediaPipe. An interpretable XGBoost-based predictive model is then developed to analyze the relationship between body motion and affective expression. Experimental results demonstrate that movements of the torso and limbs—particularly at the hips, shoulders, and elbows—are as critical as facial cues for emotion recognition, thereby challenging face-centric paradigms. The model achieves an average balanced accuracy of 0.631 across three emotion classes, substantiating the significant contribution of bodily articulation to emotional communication in sign language.

Technology Category

• Computer Vision: Biometrics, Face, Gesture & Pose
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Intelligent Robots: Embodied AI

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

We present a dataset and a model for sentiment analysis of German sign language (DGS) fairy tales. First, we perform sentiment analysis for three levels of valence (negative, neutral, positive) on German fairy tales text segments using four large language models (LLMs) and majority voting, reaching an inter-annotator agreement of 0.781 Krippendorff's alpha. Second, we extract face and body motion features from each corresponding DGS video segment using MediaPipe. Finally, we train an explainable model (based on XGBoost) to predict negative, neutral or positive sentiment from video features. Results show an average balanced accuracy of 0.631. A thorough analysis of the most important features reveal that, in addition to eyebrows and mouth motion on the face, also the motion of hips, elbows, and shoulders considerably contribute in the discrimination of the conveyed sentiment, indicating an equal importance of face and body for sentiment communication in sign language.

Problem

• Sentiment Analysis
• German Sign Language
• Facial Motion
• Body Motion
• Emotion Recognition

Innovation

• sign language sentiment analysis
• multimodal feature extraction
• explainable AI
• body motion cues
• German Sign Language (DGS)