LPIPS-AttnWav2Lip: Generic audio-driven lip synchronization for talking head generation in the wild

2023-12-01

Speech Communication

AI Summary

This work proposes a general-purpose approach to speaker-agnostic lip-sync generation by effectively integrating audiovisual features through a residual CBAM attention module embedded within a U-Net architecture. To enhance cross-modal alignment, a semantic alignment module is introduced to expand the receptive field, enabling precise matching between audio and visual representations. The method further incorporates the LPIPS perceptual loss to significantly improve the visual realism of generated faces and the consistency between audio and video. Experimental results demonstrate that the proposed method achieves state-of-the-art performance in both subjective evaluations and objective metrics, exhibiting strong generalization capabilities across unseen speakers and delivering high lip-sync accuracy and image quality.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Problem

• lip synchronization
• talking head generation
• audio-driven
• audio-visual coherence

Innovation

• Lip Synchronization
• Audio-Driven Talking Head
• CBAM Attention
• Semantic Alignment
• LPIPS Loss