AVSCap: Orchestrating Audio-Visual Synergy for Omni-modal Video Captioning

2026-07-14

AI Summary

This work addresses the challenge that existing video captioning models struggle to effectively model the co-evolution of non-speech audio and visual events. To this end, the authors propose the AVSCap framework, which first constructs a 130K-sample trimodal corpus, AVSCap-130K, via a decoupled-then-fused data curation pipeline. They further introduce a two-stage training strategy combining supervised fine-tuning with sample-efficient reinforcement learning, enhanced by a hybrid reward mechanism and explicit cross-modal event alignment. The newly released fine-grained evaluation benchmark, AVSCapBench, along with comprehensive experiments, demonstrates that AVSCap-7B significantly improves the description of non-speech sounds and the accuracy of audio-visual event association, achieving state-of-the-art performance across multiple benchmarks.

Abstract

Omni-modal video captioning is not merely combining visual captioning with audio transcription: a useful caption must describe how visual actions, speech, music, and sound effects co-evolve. Existing large multimodal models often fail at this relational step, treating audio and visual streams as loosely coupled observations, relying on automatic speech recognition, and under-specifying non-speech sounds and their links to visual events. We present AVSCap, a framework for audio-visual captioning centered on explicit cross-modal event binding. First, we construct AVSCap-130K, a tri-modal training corpus generated by a decoupled-then-fused pipeline that anchors visual and acoustic evidence before composing grounded omni-modal captions. Second, we train AVSCap-7B, a 7B captioner with a two-stage strategy: supervised fine-tuning establishes baseline capabilities, while sample-efficient reinforcement learning uses hybrid rewards to optimize acoustic completeness and audio-visual synergy. Our scaling analysis shows that reinforcement learning brings larger gains than increasing SFT data. Third, we introduce AVSCapBench, a benchmark that decomposes captions into visual, audio, and synergy events and evaluates them with fine-grained event recall. Experiments on AVSCapBench and external benchmarks show that AVSCap-7B improves non-speech audio coverage and cross-modal binding, delivering the best overall performance among evaluated open-source models.

Problem

• omni-modal video captioning
• audio-visual synergy
• cross-modal event binding
• non-speech sounds
• multimodal modeling

Innovation

• audio-visual synergy
• cross-modal event binding
• omni-modal video captioning
• reinforcement learning
• AVSCapBench