EmoHead: Emotional Talking Head via Manipulating Semantic Expression Parameters

2025-03-25

AI Summary

This work addresses the challenge of emotion-controllable audio-driven talking-head video generation, where abstract and ambiguous emotional semantics hinder precise facial expression control. We propose a label-specifiable audio-expression mapping module, integrated with a pretrained hyperplane orthogonal probing mechanism, enabling—for the first time—fine-grained, disentangled modulation of neural radiance fields (NeRFs) via interpretable emotion parameters. Our method jointly models audio and facial expressions while explicitly disentangling semantic emotion parameters, facilitating emotion-aware head animation synthesis within the NeRF rendering framework. Extensive experiments demonstrate significant improvements in expression reconstruction fidelity and emotion consistency across multi-emotion control tasks. Both visual quality and emotion controllability achieve state-of-the-art performance.

Technology Category

• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Computer Vision: Language and Vision
• Cognitive Modeling & Cognitive Systems: Affective Computing

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization

Abstract

Generating emotion-specific talking head videos from audio input is an important and complex challenge for human-machine interaction. However, emotion is highly abstract concept with ambiguous boundaries, and it necessitates disentangled expression parameters to generate emotionally expressive talking head videos. In this work, we present EmoHead to synthesize talking head videos via semantic expression parameters. To predict expression parameter for arbitrary audio input, we apply an audio-expression module that can be specified by an emotion tag. This module aims to enhance correlation from audio input across various emotions. Furthermore, we leverage pre-trained hyperplane to refine facial movements by probing along the vertical direction. Finally, the refined expression parameters regularize neural radiance fields and facilitate the emotion-consistent generation of talking head videos. Experimental results demonstrate that semantic expression parameters lead to better reconstruction quality and controllability.

Problem

• Generate emotion-specific talking head videos from audio input
• Disentangle expression parameters for emotional video synthesis
• Enhance audio-emotion correlation for consistent facial movements

Innovation

• EmoHead uses semantic expression parameters
• Audio-expression module enhances emotion correlation
• Pre-trained hyperplane refines facial movements