QMAVIS: Long Video-Audio Understanding using Fusion of Large Multimodal Models

2026-01-10

arXiv.org

AI Summary

Existing video-audio multimodal models struggle to effectively capture temporal semantics and fine-grained audio details in long-form videos spanning minutes to hours. This work proposes a novel long-form video-audio understanding framework based on late fusion with large models, introducing late fusion strategies to this task for the first time. By synergistically integrating large multimodal models (LMMs), large language models (LLMs), and automatic speech recognition (ASR) systems, the framework jointly parses narrative structures and scene-level details. It overcomes conventional limitations in video duration and semantic coherence, achieving a 38.75% performance gain over VideoLlaMA2 and InternVL2 on VideoMME (with subtitles) and up to a 2% improvement on challenging benchmarks such as PerceptionTest and EgoSchema. These advances significantly enhance long-video perception capabilities and broaden their applicability in domains like embodied intelligence.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Multimodal Models (LMMs) for video-audio understanding have traditionally been evaluated only on shorter videos of a few minutes long. In this paper, we introduce QMAVIS (Q Team-Multimodal Audio Video Intelligent Sensemaking), a novel long video-audio understanding pipeline built through a late fusion of LMMs, Large Language Models, and speech recognition models. QMAVIS addresses the gap in long-form video analytics, particularly for longer videos of a few minutes to beyond an hour long, opening up new potential applica- tions in sensemaking, video content analysis, embodied AI, etc. Quantitative experiments using QMAVIS demonstrated a 38.75% improvement over state-of-the-art video-audio LMMs like Vide- oLlaMA2 and InternVL2 on the VideoMME (with subtitles) dataset, which comprises long videos with audio information. Evaluations on other challenging video understanding datasets like PerceptionTest and EgoSchema saw up to 2% improvement, indicating competitive performance. Qualitative experiments also showed that QMAVIS is able to extract the nuances of different scenes in a long video audio content while understanding the overarching narrative. Ablation studies were also conducted to ascertain the impact of each component in the fusion pipeline.

Problem

• long video-audio understanding
• large multimodal models
• video content analysis
• sensemaking
• embodied AI

Innovation

• long video-audio understanding
• late fusion
• large multimodal models
• video content analysis
• speech recognition integration