Electromyography-Informed Facial Expression Reconstruction for Physiological-Based Synthesis and Analysis

2025-03-12

AI Summary

Existing facial expression analysis methods struggle to reconstruct authentic expressions under sEMG electrode occlusion, hindering electromyography (EMG)-vision synchronized modeling. To address this, we propose a novel EMG-guided end-to-end facial expression reconstruction paradigm: it decouples 3D geometry (based on the 3D Morphable Model) from appearance modeling and establishes a cross-modal bidirectional mapping between sEMG signals and facial action unit parameters. By integrating unpaired neural image translation, adversarial learning, and joint EMG-expression embedding, our framework enables physiology-driven expression synthesis and inverse analysis. Evaluated on a newly constructed synchronized sEMG-video dataset, our method significantly improves geometric and photorealistic reconstruction fidelity—outperforming state-of-the-art baselines both qualitatively and quantitatively. Notably, it is the first to overcome electrode occlusion constraints, enabling dynamic sEMG-based expression prediction and controllable, user-guided facial animation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The relationship between muscle activity and resulting facial expressions is crucial for various fields, including psychology, medicine, and entertainment. The synchronous recording of facial mimicry and muscular activity via surface electromyography (sEMG) provides a unique window into these complex dynamics. Unfortunately, existing methods for facial analysis cannot handle electrode occlusion, rendering them ineffective. Even with occlusion-free reference images of the same person, variations in expression intensity and execution are unmatchable. Our electromyography-informed facial expression reconstruction (EIFER) approach is a novel method to restore faces under sEMG occlusion faithfully in an adversarial manner. We decouple facial geometry and visual appearance (e.g., skin texture, lighting, electrodes) by combining a 3D Morphable Model (3DMM) with neural unpaired image-to-image translation via reference recordings. Then, EIFER learns a bidirectional mapping between 3DMM expression parameters and muscle activity, establishing correspondence between the two domains. We validate the effectiveness of our approach through experiments on a dataset of synchronized sEMG recordings and facial mimicry, demonstrating faithful geometry and appearance reconstruction. Further, we synthesize expressions based on muscle activity and how observed expressions can predict dynamic muscle activity. Consequently, EIFER introduces a new paradigm for facial electromyography, which could be extended to other forms of multi-modal face recordings.

Problem

• Reconstructs facial expressions from occluded electromyography data.
• Establishes mapping between muscle activity and 3D facial geometry.
• Enables synthesis and prediction of dynamic muscle activity.

Innovation

• Combines 3DMM with neural image translation
• Bidirectional mapping between 3DMM and sEMG
• Adversarial restoration of occluded facial expressions