Learning Spatial-Temporal Coherent Correlations for Speech-Preserving Facial Expression Manipulation

2026-04-22

AI Summary

This work addresses the challenge of editing facial expressions while accurately preserving speech-consistent lip movements in the absence of paired training data. The authors propose a spatial-temporal coherence-aware correlation learning algorithm that, for the first time, leverages the inherent spatial and temporal coherence of facial animations across different emotional expressions—when driven by identical speech content—as a self-supervised signal. To enhance performance on challenging regions, they introduce a correlation-aware region-adaptive strategy that dynamically focuses on difficult-to-edit areas. By integrating spatial-temporal coherence metrics with localized generative supervision losses, the method achieves high-fidelity expression manipulation while precisely maintaining speech-driven mouth motions, significantly improving the practical applicability of speech-driven facial expression editing models (SPFEM) in real-world scenarios.

Technology Category

• Natural Language Processing: Speech
• Machine Learning: Multimodal Learning
• Computer Vision: Language and Vision

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Speech-preserving facial expression manipulation (SPFEM) aims to modify facial emotions while meticulously maintaining the mouth animation associated with spoken content. Current works depend on inaccessible paired training samples for the person, where two aligned frames exhibit the same speech content yet differ in emotional expression, limiting the SPFEM applications in real-world scenarios. In this work, we discover that speakers who convey the same content with different emotions exhibit highly correlated local facial animations in both spatial and temporal spaces, providing valuable supervision for SPFEM. To capitalize on this insight, we propose a novel spatial-temporal coherent correlation learning (STCCL) algorithm, which models the aforementioned correlations as explicit metrics and integrates the metrics to supervise manipulating facial expression and meanwhile better preserving the facial animation of spoken content. To this end, it first learns a spatial coherent correlation metric, ensuring that the visual correlations of adjacent local regions within an image linked to a specific emotion closely resemble those of corresponding regions in an image linked to a different emotion. Simultaneously, it develops a temporal coherent correlation metric, ensuring that the visual correlations of specific regions across adjacent image frames associated with one emotion are similar to those in the corresponding regions of frames associated with another emotion. Recognizing that visual correlations are not uniform across all regions, we have also crafted a correlation-aware adaptive strategy that prioritizes regions that present greater challenges. During SPFEM model training, we construct the spatial-temporal coherent correlation metric between corresponding local regions of the input and output image frames as an additional loss to supervise the generation process.

Problem

• Speech-preserving facial expression manipulation
• paired training samples
• facial animation preservation
• emotion expression
• real-world application

Innovation

• spatial-temporal coherent correlation
• speech-preserving facial expression manipulation
• correlation-aware adaptive strategy
• facial animation preservation
• unsupervised emotion editing