A Simple Baseline for Streaming Video Understanding

2026-04-02

AI Summary

This work addresses the prevailing reliance on complex memory mechanisms in existing streaming video understanding methods, which often lack systematic evaluation against simpler baselines. The authors propose SimpleStream, a minimalist approach that feeds only the most recent N frames into an off-the-shelf vision-language model (VLM) via a sliding window, without any additional memory, retrieval, or compression modules. Experimental results demonstrate that using just four frames, SimpleStream achieves average accuracies of 67.7% on OVO-Bench and 80.59% on StreamingBench, outperforming 13 state-of-the-art methods. This study highlights a fundamental trade-off between perception and memory, arguing for a clear distinction between short-term perceptual processing and long-term memory capabilities, and establishes a new, simple yet strong baseline for streaming video understanding.

Abstract

Recent streaming video understanding methods increasingly rely on complex memory mechanisms to handle long video streams. We challenge this trend with a simple finding: a sliding-window baseline that feeds only the most recent N frames to an off-the-shelf VLM already matches or surpasses published streaming models. We formalize this baseline as SimpleStream and evaluate it against 13 major offline and online video LLM baselines on OVO-Bench and StreamingBench. Despite its simplicity, SimpleStream delivers consistently strong performance. With only 4 recent frames, it reaches 67.7% average accuracy on OVO-Bench and 80.59% on StreamingBench. Controlled ablations further show that the value of longer context is backbone-dependent rather than uniformly increasing with model scale, and reveal a consistent perception-memory trade-off: adding more historical context can improve recall, but often weakens real-time perception. This suggests that stronger memory, retrieval, or compression modules should not be taken as evidence of progress unless they clearly outperform SimpleStream under the same protocol. We therefore argue that future streaming benchmarks should separate recent-scene perception from long-range memory, so that performance improvements from added complexity can be evaluated more clearly.

Problem

• streaming video understanding
• memory mechanisms
• sliding-window baseline
• video LLM
• perception-memory trade-off

Innovation

• SimpleStream
• streaming video understanding
• sliding-window baseline
• perception-memory trade-off
• visual language model