Perceive, Query&Reason: Enhancing Video QA with Question-Guided Temporal Queries

2024-12-26

AI Summary

In long-horizon video question answering, existing methods suffer from inaccurate localization of question-relevant frames and insufficient temporal alignment between visual and linguistic modalities. To address these challenges, this paper proposes T-Former, a question-guided spatiotemporal alignment framework. Its core innovation is a question-conditioned temporal query mechanism that dynamically bridges frame-level visual perception and large language model (LLM) reasoning atop a pre-trained multimodal foundation model. By integrating temporal Transformers with cross-modal alignment techniques, T-Former enhances the model’s capacity to capture fine-grained question-frame temporal dependencies. Extensive experiments demonstrate state-of-the-art performance across multiple mainstream video QA benchmarks, significantly outperforming conventional temporal modeling approaches. Ablation studies further validate the effectiveness, robustness, and generalizability of question-driven temporal modeling.

Technology Category

• Natural Language Processing: Question Answering
• Computer Vision: Multi-modal Vision
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Video Question Answering (Video QA) is a challenging video understanding task that requires models to comprehend entire videos, identify the most relevant information based on contextual cues from a given question, and reason accurately to provide answers. Recent advancements in Multimodal Large Language Models (MLLMs) have transformed video QA by leveraging their exceptional commonsense reasoning capabilities. This progress is largely driven by the effective alignment between visual data and the language space of MLLMs. However, for video QA, an additional space-time alignment poses a considerable challenge for extracting question-relevant information across frames. In this work, we investigate diverse temporal modeling techniques to integrate with MLLMs, aiming to achieve question-guided temporal modeling that leverages pre-trained visual and textual alignment in MLLMs. We propose T-Former, a novel temporal modeling method that creates a question-guided temporal bridge between frame-wise visual perception and the reasoning capabilities of LLMs. Our evaluation across multiple video QA benchmarks demonstrates that T-Former competes favorably with existing temporal modeling approaches and aligns with recent advancements in video QA.

Problem

• Video QA
• Accuracy Improvement
• Temporal Relation Understanding

Innovation

• T-Former
• Temporal Clues
• Video Question Answering