EAD-Net: Emotion-Aware Talking Head Generation with Spatial Refinement and Temporal Coherence

2026-04-25

AI Summary

Existing methods for expressive talking head generation are limited by coarse emotion labels and struggle to simultaneously achieve accurate lip-sync and long-term temporal consistency. This work proposes EAD-Net, a novel framework that integrates fine-grained textual semantics—extracted via a large language model—as an emotion control signal within a diffusion-based generative architecture. Lip-synchronization is enforced through SyncNet supervision and TREPA alignment, while a spatiotemporal dynamic-aware (STDA) strip attention mechanism captures global spatial and temporal dynamics. Furthermore, a temporal frame reasoning module (TFRM) explicitly models inter-frame temporal coherence. Extensive experiments on the HDTF and MEAD datasets demonstrate that the proposed method significantly outperforms state-of-the-art approaches in emotion expression fidelity, lip-sync accuracy, and temporal consistency.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Emotionally talking head video generation aims to generate expressive portrait videos with accurate lip synchronization and emotional facial expressions. Current methods rely on simple emotional labels, leading to insufficient semantic information. While introducing high-level semantics enhances expressiveness, it easily causes lip-sync degradation. Furthermore, mainstream generation methods struggle to balance computational efficiency and global motion awareness in long videos and suffer from poor temporal coherence. Therefore, we propose an \textbf{E}motion-\textbf{A}ware \textbf{D}iffusion model-based \textbf{Net}work, called \textbf{EAD-Net}. We introduce SyncNet supervision and Temporal Representation Alignment (TREPA) to mitigate lip-sync degradation caused by multi-modal fusion. To model complex spatio-temporal dependencies in long video sequences, we propose a Spatio-Temporal Directional Attention (STDA) mechanism that captures global motion patterns through strip attention. Additionally, we design a Temporal Frame graph Reasoning Module (TFRM) to explicitly model temporal coherence between video frames through graph structure learning. To enhance emotional semantic control, a large language model is employed to extract textual descriptions from real videos, serving as high-level semantic guidance. Experiments on the HDTF and MEAD datasets demonstrate that our method outperforms existing methods in terms of lip-sync accuracy, temporal consistency, and emotional accuracy.

Problem

• talking head generation
• emotion-aware
• lip synchronization
• temporal coherence
• spatio-temporal modeling

Innovation

• Emotion-Aware Generation
• Temporal Coherence
• Lip-Sync Preservation
• Spatio-Temporal Attention
• Diffusion Model