GaussianEmoTalker: Real-Time Emotional Talking Head Synthesis with Audio-Driven and Blendshape-Based 3D Gaussian Splatting

2026-07-01

AI Summary

Existing methods struggle to generate expressive talking heads with controllable emotional intensity under real-time constraints. This work proposes a neutral-to-emotion residual modeling strategy: it first constructs an identity-specific neutral talking space based on Gaussian Blendshapes, then predicts emotion-induced residual deformations conditioned on audio, emotion category, and intensity encodings. To achieve this, a spatial-audio-emotion multimodal attention module is introduced, and high-fidelity rendering is realized by integrating 3D Gaussian splatting with mesh displacement cues. The proposed approach outperforms current state-of-the-art methods in terms of video quality, lip-sync accuracy, controllability of emotional expression, and real-time performance.

Abstract

Audio-driven talking head synthesis has achieved impressive progress in lip synchronization and visual quality, yet generating expressive emotional avatars with controllable intensity remains challenging, especially under real-time constraints. In this paper, we present GaussianEmoTalker, an audio-driven framework for real-time emotional talking head synthesis based on 3D Gaussian Splatting. Instead of directly predicting the final emotional avatar from speech, we formulate emotional animation as a neutral-to-emotional residual deformation problem. GaussianEmoTalker first constructs an identity-specific neutral talking space with GaussianBlendshapes, which provides high-fidelity Gaussian attributes and phoneme-synchronized neutral motion. It then predicts an emotion-conditioned residual deformation by combining mesh displacement cues, audio features, emotion categories, and intensity encodings. To fuse these heterogeneous signals, we introduce a spatial-audio-emotion attention module that estimates the offsets of Gaussian attributes for expressive and temporally stable rendering. Extensive experiments demonstrate that GaussianEmoTalker achieves competitive video quality, accurate lip synchronization, controllable emotional expression, and real-time rendering compared with recent emotional talking head methods. Our project page is available at https://njust-yang.github.io/GaussianEmoTalker.github.io/

Problem

• emotional talking head
• audio-driven synthesis
• real-time rendering
• controllable emotion
• 3D facial animation

Innovation

• 3D Gaussian Splatting
• Emotional Talking Head
• Blendshape-Based Deformation
• Residual Animation
• Real-Time Rendering