Chain-of-Glimpse: Search-Guided Progressive Object-Grounded Reasoning for Video Understanding

2026-04-16

AI Summary

Existing video understanding methods struggle to model objects undergoing semantically significant changes over time and lack explicit reasoning over critical visual evidence. This work proposes a search-guided, progressive object grounding framework that incrementally anchors task-relevant visual regions through a reinforcement learning–driven search controller coupled with a novel formatting reward mechanism. By explicitly incentivizing the model to focus on authentic visual evidence, the approach constructs spatially grounded, multi-step reasoning trajectories. Notably, it introduces formatting rewards into visual reasoning for the first time. The method consistently achieves performance gains across multiple benchmarks—including NExTQA, Video-Holmes, CG-Bench Reasoning, and VRBench—demonstrating superior accuracy, interpretability, robustness, and cross-domain generalization.

Technology Category

• Computer Vision: Visual Reasoning & Symbolic Representations
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Video understanding requires identifying and reasoning over semantically discriminative visual objects across frames, yet existing object-agnostic solutions struggle to effectively handle substantial object variations over time. To address this, we introduce Chain-of-Glimpse, a search-guided progressive object-grounded reasoning framework that explicitly anchors each reasoning step to specific visual evidence regions, enabling compositional and multi-step decision-making. Formally, Chain-of-Glimpse formulates video reasoning as a step-by-step process that incrementally builds spatially grounded traces around task-relevant visual objects, thereby mitigating over-reliance on saliency-driven cues. Specifically, Chain-of-Glimpse features a search-guided controller, optimized via reinforcement learning with a format reward that significantly incentivizes grounding capability, to iteratively ground visual evidence regions and form reliable reasoning trajectories, yielding accurate and interpretable multi-step decisions. Extensive evaluations on both in domain NExTQA and out-of-domain Video-Holmes, CG-Bench Reasoning, and VRBench benchmarks demonstrate consistent performance gains, robustness and generalization of Chain-of-Glimpse across diverse video reasoning tasks.

Problem

• video understanding
• object-grounded reasoning
• visual object variation
• multi-step reasoning
• semantic discrimination

Innovation

• object-grounded reasoning
• search-guided controller
• reinforcement learning
• visual grounding
• multi-step decision-making