CEM-Net: Cross-Emotion Memory Network for Emotional Talking Face Generation

2025-08-17

AI Summary

Existing emotional talking-face generation methods suffer from emotion misalignment and visual artifacts when reference images and driving audio convey conflicting emotions. To address this, we propose the Cross-Emotion Memory Network (CEMN), which explicitly models speech emotion via an Audio Emotion Enhancement (AEE) module and decouples identity and emotion representations from the reference image using an Emotion Bridging Memory (EBM) module—enabling controllable facial expression transfer from highly emotional references to target audio-driven emotions. CEMN further incorporates a query-based memory mechanism and cross-reconstruction training to fuse multimodal emotion features, enhancing expression consistency and naturalness. Extensive evaluations on multiple benchmarks demonstrate significant improvements in emotion accuracy, lip-sync precision, and video quality—particularly under cross-emotion conditions—outperforming prior state-of-the-art methods.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data

Abstract

Emotional talking face generation aims to animate a human face in given reference images and generate a talking video that matches the content and emotion of driving audio. However, existing methods neglect that reference images may have a strong emotion that conflicts with the audio emotion, leading to severe emotion inaccuracy and distorted generated results. To tackle the issue, we introduce a cross-emotion memory network(CEM-Net), designed to generate emotional talking faces aligned with the driving audio when reference images exhibit strong emotion. Specifically, an Audio Emotion Enhancement module(AEE) is first devised with the cross-reconstruction training strategy to enhance audio emotion, overcoming the disruption from reference image emotion. Secondly, since reference images cannot provide sufficient facial motion information of the speaker under audio emotion, an Emotion Bridging Memory module(EBM) is utilized to compensate for the lacked information. It brings in expression displacement from the reference image emotion to the audio emotion and stores it in the memory.Given a cross-emotion feature as a query, the matching displacement can be retrieved at inference time. Extensive experiments have demonstrated that our CEM-Net can synthesize expressive, natural and lip-synced talking face videos with better emotion accuracy.

Problem

• Resolves conflict between reference image and audio emotions
• Enhances audio emotion to prevent distorted results
• Compensates lacking facial motion from reference images

Innovation

• Cross-Emotion Memory Network for emotion alignment
• Audio Emotion Enhancement module for emotion clarity
• Emotion Bridging Memory compensates motion information