What You Ask is What You Ground: Bridging Question Intent to Temporal Evidence for Grounded VideoQA

2026-08-16

AI Summary

This study addresses temporal localization failures in video question answering caused by modality isolation and insufficient question injection. To overcome these limitations, we propose GroundFormer, which injects question intent prior to localization via learnable communication tokens. By integrating decomposed MIL cross-attention with Gaussian smoothing, the method achieves precise alignment of QA-aware temporal evidence, further optimized through a hierarchical multimodal contrastive loss. Experimental results demonstrate that GroundFormer attains state-of-the-art performance on both NExT-GQA and STAR datasets. The proposed approach significantly enhances discriminative question-aware temporal localization capabilities, effectively overcoming the constraints inherent in traditional methods.

Abstract

We study a critical yet overlooked failure mode in Grounded Video Question Answering: question-invariant grounding, where models predict nearly identical temporal segments for different questions about the same video. We trace this behavior to two structural limitations in prior common designs: (i) modality isolation that fixes video representations before they receive question semantics, and (ii) weak question injection inside the grounding module. To address this, we propose GroundFormer, which conditions video features on question intent before localization via learnable communication tokens that mediate directed visuo-lingual interaction. On top of the question-conditioned features, a factorized MIL cross-attention couples answer selection with temporal evidence under candidate-level supervision, while Gaussian smoothing converts peaked attention into temporally coherent segments. We further introduce a hierarchical multi-modal contrastive loss that aligns video, question, and answer embeddings across a two-pass training pipeline. GroundFormer achieves state-of-the-art grounded VideoQA performance on NExT-GQA and STAR, substantially improving question-discriminative temporal grounding.

Problem

• Grounded VideoQA
• Question-invariant grounding
• Temporal evidence localization
• Question intent

Innovation

• GroundFormer
• Question-conditioned grounding
• Factorized MIL cross-attention
• Hierarchical multi-modal contrastive loss
• Learnable communication tokens