Evaluation of Generative Models for Emotional 3D Animation Generation in VR

2025-12-17

AI Summary

This study addresses limitations of existing speech-driven 3D affective animation models in capturing subtle emotional expressions, ensuring facial/body naturalness, and delivering realistic VR interaction fidelity. Methodologically, it introduces the first user-centered evaluation conducted entirely within an immersive VR environment, integrating real-time VR interaction experiments, a five-dimensional subjective rating scale (arousal, naturalness, valence, diversity, interaction quality), reconstruction-based motion-capture baselines, and joint modeling of speech–motion synchronization with 3D facial expressions and full-body poses. Key contributions/results include: (1) establishing a novel VR-based immersive evaluation paradigm that transcends conventional 2D statistical metrics; (2) empirically demonstrating that explicit emotion modeling significantly improves emotion recognition accuracy, while revealing neutral-expression generation as a critical bottleneck; and (3) showing that joyful animations achieve highest naturalness, yet all generative models underperform the reconstruction baseline significantly in facial quality, valence, and interaction quality—though diversity is consistently rated favorably by users.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Social interactions incorporate nonverbal signals to convey emotions alongside speech, including facial expressions and body gestures. Generative models have demonstrated promising results in creating full-body nonverbal animations synchronized with speech; however, evaluations using statistical metrics in 2D settings fail to fully capture user-perceived emotions, limiting our understanding of model effectiveness. To address this, we evaluate emotional 3D animation generative models within a Virtual Reality (VR) environment, emphasizing user-centric metrics emotional arousal realism, naturalness, enjoyment, diversity, and interaction quality in a real-time human-agent interaction scenario. Through a user study (N=48), we examine perceived emotional quality for three state of the art speech-driven 3D animation methods across two emotions happiness (high arousal) and neutral (mid arousal). Additionally, we compare these generative models against real human expressions obtained via a reconstruction-based method to assess both their strengths and limitations and how closely they replicate real human facial and body expressions. Our results demonstrate that methods explicitly modeling emotions lead to higher recognition accuracy compared to those focusing solely on speech-driven synchrony. Users rated the realism and naturalness of happy animations significantly higher than those of neutral animations, highlighting the limitations of current generative models in handling subtle emotional states. Generative models underperformed compared to reconstruction-based methods in facial expression quality, and all methods received relatively low ratings for animation enjoyment and interaction quality, emphasizing the importance of incorporating user-centric evaluations into generative model development. Finally, participants positively recognized animation diversity across all generative models.

Problem

• Evaluates emotional 3D animation generative models in VR
• Assesses user-perceived emotional quality and interaction realism
• Compares generative models against real human expressions

Innovation

• Evaluated 3D animation models in VR for user-centric emotional metrics
• Compared generative models against real human expressions via reconstruction
• Found explicit emotion modeling improves recognition over speech synchrony