SynchroRaMa : Lip-Synchronized and Emotion-Aware Talking Face Generation via Multi-Modal Emotion Embedding

2025-09-24

AI Summary

Existing audio-driven talking face generation methods predominantly rely on unimodal emotion modeling and static reference images, limiting their capacity to capture subtle emotional dynamics and temporal variations. To address this, we propose a multimodal emotion embedding framework that jointly leverages text-based sentiment analysis, speech emotion recognition (using valence-arousal features), and semantically rich scene descriptions generated by large language models (LLMs), enabling emotion-aware audio-to-motion mapping. This mechanism significantly improves facial expression naturalness, lip-sync accuracy, motion diversity, and temporal coherence. Extensive evaluations on multiple benchmark datasets demonstrate state-of-the-art performance in image quality, expression preservation, and motion realism. A user study further confirms superior perceptual ratings for naturalness, motion richness, and video fluency compared to existing approaches.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Multi-modal Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Audio-driven talking face generation has received growing interest, particularly for applications requiring expressive and natural human-avatar interaction. However, most existing emotion-aware methods rely on a single modality (either audio or image) for emotion embedding, limiting their ability to capture nuanced affective cues. Additionally, most methods condition on a single reference image, restricting the model's ability to represent dynamic changes in actions or attributes across time. To address these issues, we introduce SynchroRaMa, a novel framework that integrates a multi-modal emotion embedding by combining emotional signals from text (via sentiment analysis) and audio (via speech-based emotion recognition and audio-derived valence-arousal features), enabling the generation of talking face videos with richer and more authentic emotional expressiveness and fidelity. To ensure natural head motion and accurate lip synchronization, SynchroRaMa includes an audio-to-motion (A2M) module that generates motion frames aligned with the input audio. Finally, SynchroRaMa incorporates scene descriptions generated by Large Language Model (LLM) as additional textual input, enabling it to capture dynamic actions and high-level semantic attributes. Conditioning the model on both visual and textual cues enhances temporal consistency and visual realism. Quantitative and qualitative experiments on benchmark datasets demonstrate that SynchroRaMa outperforms the state-of-the-art, achieving improvements in image quality, expression preservation, and motion realism. A user study further confirms that SynchroRaMa achieves higher subjective ratings than competing methods in overall naturalness, motion diversity, and video smoothness. Our project page is available at <https://novicemm.github.io/synchrorama>.

Problem

• Generating talking faces with synchronized lip movements and emotional expressiveness
• Overcoming limitations of single-modality emotion embedding in existing methods
• Enhancing dynamic action representation beyond static reference images

Innovation

• Multi-modal emotion embedding from text and audio
• Audio-to-motion module for lip synchronization
• LLM-generated scene descriptions for dynamic actions