Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning

2026-07-01

AI Summary

This work addresses the limitations of existing audio-visual captioning methods, which often fail to accurately associate auditory events with visual entities or model complex causal dynamics due to modality misalignment and temporal inconsistency. To overcome these challenges, the authors propose TCA-Captioner, a novel framework featuring an Observer-Checker-Corrector (OCC) iterative refinement mechanism. It leverages high-density human-machine interaction data to generate high-fidelity training samples and employs a multimodal large language model to enhance cross-modal alignment. Additionally, the study introduces TCA-Bench, a diagnostic benchmark with a disentangled evaluation protocol that separately quantifies a model’s capabilities in audio-visual binding and temporal reasoning. Experiments demonstrate that the proposed approach significantly improves temporal coherence and cross-modal synchronization in generated captions, establishing a new state-of-the-art on TCA-Bench.

Abstract

While Multimodal Large Language Models (MLLMs) have advanced video understanding, achieving precise temporal and cross-modal alignment in audiovisual video captioning remains a formidable challenge. Most existing approaches suffer from modality detachment and temporal incoherence, failing to accurately bind auditory events to visual entities or capture complex causal dynamics. To address these deficiencies, we propose TCA-Captioner, a framework specifically engineered to enhance Temporal and Cross-Modal Alignment for audiovisual video captioning. We first introduce the Observer-Checker-Corrector (OCC) framework, an iterative refinement strategy that generates high-fidelity, meticulously grounded training data. Leveraging a curated high-density human interaction dataset, TCA-Captioner is optimized to model sophisticated audiovisual interactions. Furthermore, we present TCA-Bench, a diagnostic benchmark utilizing a Decoupled Evaluation Protocol to isolate and quantify model proficiency in audiovisual binding and temporal relational reasoning. Extensive experiments demonstrate that TCA-Captioner sets a new standard for temporally-coherent and synchronized audiovisual narratives.

Problem

• Temporal Alignment
• Cross-Modal Alignment
• Audiovisual Video Captioning
• Modality Detachment
• Temporal Incoherence

Innovation

• Temporal Alignment
• Cross-Modal Alignment
• Audiovisual Captioning
• Iterative Refinement
• Decoupled Evaluation