🤖 AI Summary
Existing video-to-audio generation methods suffer from imprecise visual-audio temporal and semantic alignment, primarily due to the lack of fine-grained alignment annotations in open benchmark datasets. To address this, we propose a novel inference-guided generation framework that requires no manual annotation: (1) we pioneer the integration of chain-of-thought (CoT) reasoning from multimodal large language models (MLLMs) directly into the audio synthesis pipeline, enabling stepwise, interpretable cross-modal alignment; (2) we construct the first video-audio-text multimodal reasoning dataset explicitly designed for inference-guided generation; and (3) we jointly optimize self-supervised temporal reasoning and cross-modal alignment modeling. Experiments demonstrate significant improvements: a 0.89% reduction in voice misalignment rate, and decreases of 10.07% in FDPaSST, 11.62% in FDPA-NNs, and 38.61% in FDVGG—alongside gains of 4.95% in Inception Score (IS) and 6.39% in IB-score—establishing new state-of-the-art performance.
📝 Abstract
Currently, high-quality, synchronized audio is synthesized from video and optional text inputs using various multi-modal joint learning frameworks. However, the precise alignment between the visual and generated audio domains remains far from satisfactory. One key factor is the lack of sufficient temporal and semantic alignment annotations in open-source video-audio and text-audio benchmarks. Therefore, we propose a framework for audio generation from videos, leveraging the internal chain-of-thought (CoT) of a multi-modal large language model (MLLM) to enable step-by-step reasoning without requiring additional annotations. Additionally, a corresponding multi-modal reasoning dataset is constructed to facilitate the learning of initial reasoning in audio generation. In the experiments, we demonstrate the effectiveness of the proposed framework in reducing misalignment (voice-over) in generated audio and achieving competitive performance compared to various state-of-the-art models. The evaluation results show that the proposed method outperforms state-of-the-art approaches across multiple metrics. Specifically, the F DP aSST indicator is reduced by up to 10.07%, the F DP AN N s indicator by up to 11.62%, and the F DV GG indicator by up to 38.61%. Furthermore, the IS indicator improves by up to 4.95%, the IB-score indicator increases by up to 6.39%, and the DeSync indicator is reduced by up to 0.89%.