Team of One: Cracking Complex Video QA with Model Synergy

2025-07-18

AI Summary

Existing video large multimodal models (Video-LMMs) suffer from shallow contextual understanding, weak temporal modeling, and poor generalization to ambiguous or compositional queries in complex video question answering. To address these limitations, we propose a fine-tuning-free, multi-model collaborative reasoning framework: heterogeneous video-language models (VLMs) concurrently generate structured chain-of-thought (CoT) reasoning paths, which are dynamically evaluated, selected, and fused by an external large language model (LLM). We introduce the first prompting-and-response integration mechanism, enabling a lightweight and scalable multimodal collaboration architecture. Evaluated on the CVRR-ES benchmark, our approach comprehensively outperforms state-of-the-art methods, achieving significant gains in accuracy, robustness, and generalization. This work establishes an efficient, scalable new paradigm for complex video understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

We propose a novel framework for open-ended video question answering that enhances reasoning depth and robustness in complex real-world scenarios, as benchmarked on the CVRR-ES dataset. Existing Video-Large Multimodal Models (Video-LMMs) often exhibit limited contextual understanding, weak temporal modeling, and poor generalization to ambiguous or compositional queries. To address these challenges, we introduce a prompting-and-response integration mechanism that coordinates multiple heterogeneous Video-Language Models (VLMs) via structured chains of thought, each tailored to distinct reasoning pathways. An external Large Language Model (LLM) serves as an evaluator and integrator, selecting and fusing the most reliable responses. Extensive experiments demonstrate that our method significantly outperforms existing baselines across all evaluation metrics, showcasing superior generalization and robustness. Our approach offers a lightweight, extensible strategy for advancing multimodal reasoning without requiring model retraining, setting a strong foundation for future Video-LMM development.

Problem

• Enhancing reasoning depth in video question answering
• Addressing limited contextual understanding in Video-LMMs
• Improving generalization to ambiguous or compositional queries

Innovation

• Prompting-and-response integration mechanism for VLMs
• LLM as evaluator and integrator for reliable responses
• Lightweight, extensible strategy without model retraining