Not All Modalities Are Equal: Instruction-Aware Gating for Multimodal Videos

2026-05-25

AI Summary

This work addresses the challenge in multimodal video understanding where naive fusion of auxiliary modalities—such as audio or depth maps—often introduces interference and diverts model attention from critical information. To this end, the authors propose UniMVU, a novel framework featuring the first instruction-aware, dual-level dynamic gating mechanism. Specifically, intra-modality gating highlights salient regions within each modality, while inter-modality gating adaptively reweights modality streams based on textual instructions. Coupled with a fast-slow fusion strategy to reduce temporal redundancy, this approach enables unified, scalable, and interpretable multimodal integration without handcrafted rules. Evaluated across six benchmarks—including AVQA and AVSD—the method substantially outperforms static fusion baselines, achieving up to a 13.5-point gain in CIDEr score.

Abstract

Pre-trained video large language models excel at visual reasoning. However, they struggle when videos arrive with auxiliary streams, such as audio, depth map, or dense temporal evidence. In such a scenario, uniform fusion induces modality interference, allowing irrelevant channels to distract the model. To address this issue, we present a unified multimodal video understanding framework, named UniMVU, that performs instruction-aware fusion across video, audio, depth map, or any other modality inputs via two levels of dynamic gating: inner-modality gates emphasize salient regions within each modality, whereas modality-level gates re-weight whole streams; both are conditioned on the text instruction to adaptively balance modality importance. Our UniMVU combines cross-modal self-attention with instruction-driven inner-modality gating module and a modality-level gating module with control token; for time-aligned streams we further adopt a fast-to-slow fusion scheme that reduces redundancy. Across six benchmarks (AVQA, AVSD, Music-AVQA, ScanQA, SQA3D and MVBench), our UniMVU achieves consistent gains over static-fusion baselines achieving gains as high as 13.5 in terms of CIDEr metric. Further, our analysis shows that the gating mechanism aligns with the human-interpretable modality relevance, and ablations show the contributions of inner-modality and modality-level gating. Our UniMVU provides a simple, unified recipe for instruction-aware multimodal video understanding that scales to diverse modalities without hand-crafted fusion rules.

Problem

• multimodal video understanding
• modality interference
• instruction-aware fusion
• auxiliary modalities
• video large language models

Innovation

• instruction-aware gating
• multimodal video understanding
• dynamic gating
• modality interference
• fast-to-slow fusion