Response-G1: Explicit Scene Graph Modeling for Proactive Streaming Video Understanding

2026-05-08

AI Summary

This work addresses the challenge in streaming video understanding where models struggle to determine the optimal response timing due to limitations in existing approaches that rely on implicit and query-agnostic modeling of visual evidence. To overcome this, the paper introduces explicit scene graphs for the first time and proposes a novel framework comprising query-guided online scene graph generation, memory-based semantic retrieval, and a retrieval-augmented trigger prompting mechanism. This enables interpretable, fine-tuning-free response decisions by aligning video evidence with query conditions through structured graph representations. The method significantly outperforms prior art across multiple benchmarks and achieves state-of-the-art performance in both active and passive video understanding tasks.

Abstract

Proactive streaming video understanding requires Video-LLMs to decide when to respond as a video unfolds, a task where existing methods often fall short due to their implicit, query-agnostic modeling of visual evidence. We introduce Response-G1, a novel framework that establishes explicit, structured alignment between the accumulated video evidence and the query's expected response conditions via scene graphs. The framework operates in three fine-tuning-free stages: (1) online query-guided scene graph generation from streaming clips; (2) memory-based retrieval of the most semantically relevant historical scene graphs; and (3) retrieval-augmented trigger prompting for per-frame "silence/response" decisions.By grounding both evidence and conditions in a shared graph representation, Response-G1 achieves more interpretable and accurate response timing decisions. Experimental results on established benchmarks demonstrate the superiority of our method in both proactive and reactive tasks, validating the advantage of explicit scene graph modeling and retrieval in streaming video understanding.

Problem

• proactive streaming video understanding
• Video-LLMs
• scene graph modeling
• response timing
• visual evidence

Innovation

• scene graph
• proactive video understanding
• retrieval-augmented prompting
• streaming video
• Video-LLM