Don't Pause! Every prediction matters in a streaming video

2026-04-27

AI Summary

Existing online video question-answering benchmarks predominantly employ retrospective evaluation, which fails to assess a model’s ability to respond in real time to events in streaming video. To address this limitation, this work introduces SPOT-Bench, the first benchmark tailored for proactive, multi-turn querying in streaming video understanding, and proposes the Timeliness-F1 metric to quantitatively evaluate model timeliness. The study further identifies the “dead-zone” phenomenon—periods during which models remain unresponsive despite relevant visual input—and designs AsynKV, a training-free mechanism that enhances offline models’ streaming inference behavior through short- and long-term memory integration and dead-zone-aware scheduling. Experiments demonstrate that AsynKV significantly outperforms existing approaches on SPOT-Bench while maintaining state-of-the-art performance on conventional retrospective benchmarks.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Data Mining & Knowledge Management: Intelligent Query Processing

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Streaming video models should respond the moment an event unfolds, not after the moment has passed. Yet existing online VideoQA benchmarks remain largely retrospective. They pause the video at fixed timestamps, pose questions about current or past events, and score models only at those moments. This protocol leaves streaming predictions untested. To close this gap, we introduce SPOT-Bench, featuring multi-turn proactive queries that evaluate general streaming perception and assistive capabilities required by an always-on, real-time assistant. SPOT-Bench comes with Timeliness-F1, a consolidated metric that measures streaming predictions by their temporal precision and balanced coverage across the entire video. Our benchmark reveals: (i) offline models detect events reliably but spam predictions unprompted; (ii) post-training for silence reduces spamming but induces unresponsiveness; (iii) half of the streaming video expects no response, which we term dead-time - compute spent here does not affect response latency. These findings motivate AsynKV, a training-free streaming adaptation of offline models, that retains their event perception while improving their streaming behavior. AsynKV features a long-short term memory, utilized efficiently by scaling compute during dead-time. It serves as a strong baseline on SPOT-Bench, outperforming existing streaming models, and achieves state-of-the-art on retrospective benchmarks.

Problem

• streaming video
• VideoQA
• real-time prediction
• online evaluation
• temporal precision

Innovation

• streaming video understanding
• proactive VideoQA
• Timeliness-F1
• dead-time optimization
• training-free adaptation