🤖 AI Summary
To address insufficient dialogue/narration/monologue style adaptation and weak modeling of fine-grained speaker attributes (e.g., age, gender) and contextual styles in film dubbing, this paper proposes a vision-driven multimodal large language model framework, introducing for the first time a chain-of-thought (CoT)-guided dubbing style perception and generation paradigm. Our method integrates vision–speech cross-modal CoT reasoning, a conditional large speech synthesis model (variants of VITS/Grad-TTS), and fine-grained speaker representation learning. Key contributions include: (1) releasing CoT-Movie-Dubbing—the first film dubbing dataset with CoT annotations; and (2) achieving breakthroughs in dubbing-type adaptation and joint emotion–timbre modeling. Extensive experiments on V2C, Grid, and CoT-Movie-Dubbing demonstrate state-of-the-art performance: +19.39% SPK-SIM, +12.64% EMO-SIM, −29.49 percentage points WER reduction, and significant improvements in LSE-D and MCD-SL.
📝 Abstract
Current movie dubbing technology can generate the desired voice from a given speech prompt, ensuring good synchronization between speech and visuals while accurately conveying the intended emotions. However, in movie dubbing, key aspects such as adapting to different dubbing styles, handling dialogue, narration, and monologue effectively, and understanding subtle details like the age and gender of speakers, have not been well studied. To address this challenge, we propose a framework of multi-modal large language model. First, it utilizes multimodal Chain-of-Thought (CoT) reasoning methods on visual inputs to understand dubbing styles and fine-grained attributes. Second, it generates high-quality dubbing through large speech generation models, guided by multimodal conditions. Additionally, we have developed a movie dubbing dataset with CoT annotations. The evaluation results demonstrate a performance improvement over state-of-the-art methods across multiple datasets. In particular, for the evaluation metrics, the SPK-SIM and EMO-SIM increases from 82.48% to 89.74%, 66.24% to 78.88% for dubbing setting 2.0 on V2C Animation dataset, LSE-D and MCD-SL decreases from 14.79 to 14.63, 5.24 to 4.74 for dubbing setting 2.0 on Grid dataset, SPK-SIM increases from 64.03 to 83.42 and WER decreases from 52.69% to 23.20% for initial reasoning setting on proposed CoT-Movie-Dubbing dataset in the comparison with the state-of-the art models.