Personalized Cross-Modal Emotional Correlation Learning for Speech-Preserving Facial Expression Manipulation

2026-04-28

AI Summary

This work addresses the challenge in speech-preserving facial expression manipulation caused by the scarcity of paired data featuring the same speaker uttering identical speech content with varying facial expressions, which hinders effective supervision. To overcome this limitation, the authors propose a personalized cross-modal affective association learning algorithm. The method first generates personalized prompts based on individual visual characteristics and then employs a cross-modal feature differencing alignment mechanism to precisely model fine-grained associations between speech semantics and facial expression dynamics. Built upon a vision-language foundation model, the approach adopts a plug-in design that requires no modification to the backbone architecture. Experimental results demonstrate that the proposed method significantly enhances both the naturalness and emotional fidelity of facial expression manipulation across multiple benchmark datasets.

Abstract

Speech-preserving facial expression manipulation (SPFEM) aims to enhance human expressiveness without altering mouth movements tied to the original speech. A primary challenge in this domain is the scarcity of paired data, namely aligned frames of the same individual with identical speech but different expressions, which impedes direct supervision for emotional manipulation. While current Visual-Language Models (VLMs) can extract aligned visual and semantic features, making them a promising source of supervision, their direct application is limited. To this end, we propose a Personalized Cross-Modal Emotional Correlation Learning (PCMECL) algorithm that refines VLM-based supervision through two major improvements. First, standard VLMs rely on a single generic prompt for each emotion, failing to capture expressive variations among individuals. PCMECL addresses this limitation by conditioning on individual visual information to learn personalized prompts, thereby establishing more fine-grained visual-semantic correlations. Second, even with personalization, inherent discrepancies persist between the visual and semantic feature distributions. To bridge this modality gap, PCMECL employs feature differencing to correlate the modalities, providing more precisely aligned supervision by matching the change in visual features to the change in semantic features. As a plug-and-play module, PCMECL can be seamlessly integrated into existing SPFEM models. Extensive experiments across various datasets demonstrate the superior efficacy of our algorithm.

Problem

• Speech-Preserving Facial Expression Manipulation
• Paired Data Scarcity
• Emotional Manipulation
• Cross-Modal Supervision
• Visual-Language Models

Innovation

• Personalized Prompting
• Cross-Modal Alignment
• Feature Differencing
• Visual-Language Models
• Speech-Preserving Facial Expression Manipulation