Empowering Long-form Omni-modal Understanding with Robust Audio Perception

2026-07-11

AI Summary

This work addresses the limitation of existing large-scale multimodal models in audio-visual joint understanding, which stems from the scarcity of explicitly aligned auditory annotations. To overcome this, the authors introduce the AVDC dataset, generated via an automated pipeline that produces decoupled captions for visual, auditory, and audio-visual modalities, along with a chain-of-thought-enhanced question-answering benchmark, AVDC-QA-CoT. The proposed approach employs a two-stage training strategy—combining multimodal pretraining and instruction fine-tuning—to enable, for the first time, systematic fine-grained audio perception and cross-modal reasoning. Experimental results demonstrate that the framework significantly outperforms baseline models across video captioning, audio analysis, and full-modality comprehension tasks, thereby validating the efficacy of the proposed data structure and training paradigm.

Abstract

Recent advances in large-scale multimodal models have drivenremarkable progress in vision-language tasks; however, comprehensiveomni-modal understanding remains under-explored, largely due to thescarcity of datasets with rich, explicitly aligned auditory cues. To bridgethis gap, we present AVDC (Audio-Visual Decoupled Captions), a large-scaledataset designed to disentangle visual and auditory semantics. Specifi-cally, we propose an automated pipeline that leverages off-the-shelf mod-els to annotate videos with tripartite captions: visual-only (V), audio-only (A), and joint audio-visual (AV). This decoupled structure explic-itly captures both modality-specific nuances and complex cross-modalinteractions. Building upon this, we introduce AVDC-QA-CoT, a Chain-of-Thought augmented question-answering dataset to foster audio-visualreasoning. To fully exploit these resources, we employ a two-stage train-ing paradigm: omni-modal caption generation pre-training on AVDC, fol-lowed by instruction tuning on AVDC-QA-CoT. Extensive experiments acrossdiverse downstream tasks, spanning video captioning, audio-centric anal-ysis, and omni-modal benchmarks, demonstrate consistent and signifi-cant performance gains, showing the efficacy of our proposed datasetsand training strategy in advancing omni-modal perception. Code anddataset are related on https://radiant0726.github.io/AVDC-web/.

Problem

• omni-modal understanding
• audio perception
• multimodal datasets
• audio-visual alignment
• data scarcity

Innovation

• omni-modal understanding
• audio-visual decoupling
• Chain-of-Thought reasoning
• multimodal pre-training
• AVDC dataset