CoSyncDiT: Cognitive Synchronous Diffusion Transformer for Movie Dubbing

2026-04-14

AI Summary

This work addresses the longstanding challenge in movie dubbing of simultaneously achieving accurate lip-sync and natural-sounding speech, which is often compromised by interference from reference audio. To this end, the authors propose a cognitive-synchronized diffusion Transformer framework that emulates the cognitive process of professional voice actors. The approach guides the denoising trajectory from noise to speech through acoustic style adaptation, fine-grained visual calibration, and temporal-aware contextual alignment. A novel cognition-inspired generation pipeline is introduced, complemented by a Joint Semantic and Alignment Regularization (JSAR) mechanism that concurrently enforces frame-level temporal consistency and semantic coherence in streaming hidden states. Built upon a flow-matching-based diffusion Transformer architecture, the method achieves state-of-the-art performance on both standard and real-world dubbing benchmarks, significantly improving both lip-sync accuracy and speech naturalness.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Speech
• Cognitive Modeling & Cognitive Systems: Adaptive Behavior

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Movie dubbing aims to synthesize speech that preserves the vocal identity of a reference audio while synchronizing with the lip movements in a target video. Existing methods fail to achieve precise lip-sync and lack naturalness due to explicit alignment at the duration level. While implicit alignment solutions have emerged, they remain susceptible to interference from the reference audio, triggering timbre and pronunciation degradation in in-the-wild scenarios. In this paper, we propose a novel flow matching-based movie dubbing framework driven by the Cognitive Synchronous Diffusion Transformer (CoSync-DiT), inspired by the cognitive process of professional actors. This architecture progressively guides the noise-to-speech generative trajectory by executing acoustic style adapting, fine-grained visual calibrating, and time-aware context aligning. Furthermore, we design the Joint Semantic and Alignment Regularization (JSAR) mechanism to simultaneously constrain frame-level temporal consistency on the contextual outputs and semantic consistency on the flow hidden states, ensuring robust alignment. Extensive experiments on both standard benchmarks and challenging in-the-wild dubbing benchmarks demonstrate that our method achieves the state-of-the-art performance across multiple metrics.

Problem

• movie dubbing
• lip-sync
• speech synthesis
• reference audio interference
• naturalness

Innovation

• Cognitive Synchronous Diffusion Transformer
• flow matching
• lip-sync
• Joint Semantic and Alignment Regularization
• movie dubbing