ViTexQA: A Multi-Frame Temporal Perception Dataset for Video Text Question Answering

2026-06-23

AI Summary

This work addresses the limitation of existing video-text understanding benchmarks, which predominantly rely on single-frame information and thus fail to support complex reasoning requiring cross-frame temporal textual cues. To bridge this gap, the authors introduce ViTexQA, the first video question-answering dataset that explicitly necessitates multi-frame text fusion. They further propose FrameThinker, a novel model trained via a two-stage framework: first, chain-of-thought (CoT)-guided supervised fine-tuning generates frame-aware reasoning chains; second, temporal alignment–aware reinforcement learning, augmented with a multi-frame consistency reward mechanism, refines temporal reasoning capabilities. Experimental results demonstrate that FrameThinker substantially outperforms state-of-the-art models on ViTexQA, achieving a 6.3% absolute improvement in ROUGE-L score, thereby advancing multi-frame temporal modeling in video-text understanding.

Abstract

Despite remarkable progress in multimodal understanding, current MLLMs still exhibit limitations in video text understanding, particularly when semantics emerge through the integration of temporally distributed textual cues across multiple frames. This perception challenge fundamentally differs from static image text understanding, yet existing datasets fail to capture: the vast majority of questions remain answerable from single frames, inadequately reflecting real-world video text comprehension demands. To address this, we present ViTexQA, a large-scale video-text QA dataset, and FrameThinker for robust multi-frame temporal reasoning. We build ViTexQA via a quality-controlled Chain-of-Thought (CoT) annotation pipeline boosted with temporal constraints; all its QA pairs demand cross-frame text fusion to solve, enforcing true temporal reliance. FrameThinker adopts two-stage training for explicit temporal modeling: CoT-Guided Supervised Fine-Tuning (SFT) generates frame-aware reasoning chains, followed by Temporally-grounded Reinforcement Learning (RL) optimized with multi-frame coherence rewards. Evaluations show our method outperforms SOTA baselines on ViTexQA, lifting ROUGE-L by 6.3%.

Problem

• video text understanding
• temporal perception
• multi-frame reasoning
• multimodal QA
• temporal coherence

Innovation

• multi-frame temporal reasoning
• video text question answering
• Chain-of-Thought annotation
• temporally-grounded reinforcement learning
• cross-frame text fusion