What Should a Streaming Video Model Remember?

2026-06-15

AI Summary

This work addresses the challenge of enabling streaming video models to answer queries at arbitrary timestamps under fixed memory and computation budgets, while preventing interference from historical information with current perception. The authors propose SelectStream, a framework that formulates memory management as a budget-constrained online latent evidence allocation problem. SelectStream achieves selective memory by integrating surprise-driven adaptive windowing, priority-preserving compression, and query-conditioned graph reasoning, thereby incorporating only compact, query-relevant historical evidence while preserving direct access to the current frame. Built upon a frozen vision-language model, the method requires neither frame replay nor context that scales with stream length. It significantly outperforms existing streaming and window-based baselines on StreamingBench (82.67%), OVO-Bench (67.03%), and offline video benchmarks (average 74.4%).

Abstract

Streaming video understanding models must answer queries at any moment during an ongoing stream, using only what they have observed so far and under fixed memory and computation budgets. Existing methods address this by adding memory banks, retrieval modules, or visual token compression to preserve long-range history. However, strong recent-window baselines show that indiscriminate history injection can dilute current-scene perception, suggesting that the key challenge is not whether to use memory, but how to allocate it selectively. We formulate this as budgeted online latent evidence allocation and propose \textbf{SelectStream}, a selective latent-memory framework that keeps the current observation directly visible to a frozen VLM while exposing historical information only through a compact, query-conditioned evidence budget. Three coordinated mechanisms govern when to write, what to preserve, and how to retrieve: surprise-driven adaptive windowing, priority-preserving consolidation, and query-conditioned graph reasoning over a fixed-capacity latent memory graph. Retrieved evidence is calibrated and injected as latent tokens for answer generation, without replaying frames or growing the context with stream length. Experimental results show that SelectStream achieves strong online streaming performance and preserves general video understanding, reaching 82.67\% on StreamingBench, 67.03\% on OVO-Bench, and 74.4\% average accuracy on offline video benchmarks, while outperforming strong recent-window baselines and prior streaming memory methods.

Problem

• streaming video understanding
• memory allocation
• selective memory
• online inference
• video question answering

Innovation

• selective memory
• streaming video understanding
• latent evidence allocation
• query-conditioned retrieval
• fixed-budget online learning