Self-ReS: Self-Reflection in Large Vision-Language Models for Long Video Understanding

2025-03-26

AI Summary

Large Vision-Language Models (LVLMs) suffer from suboptimal long-video understanding due to linear frame sampling, which often misses critical events or introduces redundancy. To address this, we propose a self-reflective nonlinear spatiotemporal sampling method that leverages LVLMs’ intrinsic sparse attention maps to generate “reflection tokens”—enabling prompt-driven, dynamic selection of key video segments without additional training, parameters, or modules. Our approach integrates self-reflection sampling, sparse-attention guidance, and prompt-conditioned spatiotemporal selection, supporting zero-shot integrative adaptation. Evaluated across multiple long-video understanding benchmarks, it achieves substantial accuracy gains while accelerating inference by up to 46% and maintaining efficiency under fixed GPU memory constraints. The core innovation lies in repurposing the LVLM’s native attention mechanism as an interpretable, controllable sampling signal—establishing a lightweight, general-purpose, plug-and-play paradigm for long-video understanding.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Large Vision-Language Models (LVLMs) demonstrate remarkable performance in short-video tasks such as video question answering, but struggle in long-video understanding. The linear frame sampling strategy, conventionally used by LVLMs, fails to account for the non-linear distribution of key events in video data, often introducing redundant or irrelevant information in longer contexts while risking the omission of critical events in shorter ones. To address this, we propose SelfReS, a non-linear spatiotemporal self-reflective sampling method that dynamically selects key video fragments based on user prompts. Unlike prior approaches, SelfReS leverages the inherently sparse attention maps of LVLMs to define reflection tokens, enabling relevance-aware token selection without requiring additional training or external modules. Experiments demonstrate that SelfReS can be seamlessly integrated into strong base LVLMs, improving long-video task accuracy and achieving up to 46% faster inference speed within the same GPU memory budget.

Problem

• LVLMs struggle with long-video understanding due to linear frame sampling
• Linear sampling risks omitting critical events and adding irrelevant information
• SelfReS dynamically selects key video fragments using sparse attention maps

Innovation

• Dynamic spatiotemporal self-reflective sampling method
• Leverages sparse attention maps for relevance
• No extra training or external modules needed