SER: Learning to Ground Video Reasoning with Semantic Evidence Rewards

2026-06-23

AI Summary

This work addresses the limitation of existing video multimodal large language models, which often rely on irrelevant frames or objects for fine-grained spatiotemporal reasoning due to a lack of reliable semantic alignment evidence. To this end, the authors reformulate spatiotemporal evidence localization as a constrained verification task and introduce a Semantic Evidence Reward (SER) mechanism. This mechanism employs a referee vision-language model (VLM) to assess the relevance and localization quality of generated evidence, augmented with a temporal consistency penalty. Notably, the approach is trained solely on standard video question-answering data without requiring dense bounding box annotations. By replacing pixel-level overlap with semantic alignment as the evaluation criterion, the method significantly enhances both interpretability and accuracy, achieving 49.6% mLGM on the V-STAR benchmark—3.0 percentage points higher than the strong baseline Open-o3-Video.

Abstract

Video MLLMs often struggle with fine-grained spatio-temporal reasoning, sometimes generating correct answers based on irrelevant frames or objects. Although outputting spatio-temporal evidence during reasoning is a promising direction, existing RL frameworks typically rely on geometry-only (IoU) rewards, which can be sensitive to boundary perturbations and overlook semantic alignment. To address this, we propose Semantic Evidence Reward (SER), which reformulates spatio-temporal evidence grounding as a constrained verification task. Instead of computing pixel-level overlap, SER uses a referee VLM as a local checker to evaluate model-generated evidence claims across two dimensions: relevance and localization quality, combined with a temporal penalty. This design reduces the reliance on dense box annotations and enables training directly on standard video QA data. On the V-STAR benchmark, SER achieves 49.6% mLGM, improving by 3.0 points over the strong evidence-grounded baseline Open-o3-Video, demonstrating its potential in enhancing both answer accuracy and evidence grounding.

Problem

• spatio-temporal reasoning
• semantic alignment
• evidence grounding
• video MLLMs
• reward design

Innovation

• Semantic Evidence Reward
• Video MLLMs
• Spatio-temporal Reasoning
• Reinforcement Learning
• Evidence Grounding