Bidirectional Learned Facial Animation Codec for Low Bitrate Talking Head Videos

2025-03-12

AI Summary

Existing talking-head video coding methods rely on single-keyframe reconstruction, limiting their ability to model large-angle head motions and resulting in facial distortions and motion discontinuities. To address this, we propose Bi-AnimCodec, the first bidirectional learning animation codec featuring a novel bidirectional keyframe adaptive selection mechanism that jointly leverages past and future keyframes to drive animation generation. We further introduce the BRG-ASE auxiliary stream enhancement module and the BRG-VRec joint reconstruction module, enabling keypoint-guided, low-bitrate, high-fidelity facial animation synthesis. The framework integrates deep generative modeling, bidirectional temporal referencing, auxiliary stream coding, and adaptive frame composition. Experiments demonstrate that Bi-AnimCodec achieves a 55% bitrate reduction over state-of-the-art animation codecs and a 35% reduction over VVC at comparable quality, while significantly improving facial detail fidelity and temporal motion coherence.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Existing deep facial animation coding techniques efficiently compress talking head videos by applying deep generative models. Instead of compressing the entire video sequence, these methods focus on compressing only the keyframe and the keypoints of non-keyframes (target frames). The target frames are then reconstructed by utilizing a single keyframe, and the keypoints of the target frame. Although these unidirectional methods can reduce the bitrate, they rely on a single keyframe and often struggle to capture large head movements accurately, resulting in distortions in the facial region. In this paper, we propose a novel bidirectional learned animation codec that generates natural facial videos using past and future keyframes. First, in the Bidirectional Reference-Guided Auxiliary Stream Enhancement (BRG-ASE) process, we introduce a compact auxiliary stream for non-keyframes, which is enhanced by adaptively selecting one of two keyframes (past and future). This stream improves video quality with a slight increase in bitrate. Then, in the Bidirectional Reference-Guided Video Reconstruction (BRG-VRec) process, we animate the adaptively selected keyframe and reconstruct the target frame using both the animated keyframe and the auxiliary frame. Extensive experiments demonstrate a 55% bitrate reduction compared to the latest animation based video codec, and a 35% bitrate reduction compared to the latest video coding standard, Versatile Video Coding (VVC) on a talking head video dataset. It showcases the efficiency of our approach in improving video quality while simultaneously decreasing bitrate.

Problem

• Improves video quality with reduced bitrate.
• Captures large head movements accurately.
• Uses past and future keyframes for reconstruction.

Innovation

• Bidirectional learned animation codec for video compression
• Adaptive keyframe selection enhances video quality
• Significant bitrate reduction compared to existing standards