ProtoKV: Streaming Video Understanding under Delayed Query with Summary-State Memory

2026-06-25

AI Summary

This work addresses the challenge in streaming video understanding where delayed queries cause critical visual cues to be overwritten or diluted under limited GPU memory. To mitigate this, the authors propose ProtoKV, a memory architecture that maintains exact key-value (KV) caches in a recent window while compressing distant historical information into a fixed-capacity semantic-spatial prototype bank. This prototype bank is seamlessly integrated into the standard attention mechanism via pseudo-token interfaces. By preserving long-term semantic information in a compact, resident summary state, ProtoKV substantially alleviates performance degradation under query latency. Experiments demonstrate that, within identical computational and memory budgets, the method improves accuracy by up to 12.5 percentage points in high-latency scenarios, with gains consistently increasing as delay grows.

Abstract

Streaming video understanding (SVU) must answer queries that arrive asynchronously while visual tokens stream continuously under strict GPU-memory and query-time latency budgets. A key challenge is delayed query: decisive cues may appear briefly, yet many subsequent updates occur before the query arrives, increasing the risk that those cues are evicted or diluted under bounded memory. We propose ProtoKV, a constant-footprint SVU memory that represents far history as a fixed-capacity summary state rather than retaining token instances. ProtoKV keeps an exact near-window KV cache and aggregates older content into a semantic-spatial prototype bank with residual statistics. At query time, each prototype is exposed through a bounded pseudo-token interface that is drop-in compatible with standard attention. Under matched budgets and comparable query-time cost, ProtoKV improves accuracy by up to 12.5 points over token-retention baselines on SVU benchmarks in the long-delay regime, with gains that grow as query delay increases.

Problem

• streaming video understanding
• delayed query
• memory constraint
• visual tokens
• query latency

Innovation

• ProtoKV
• streaming video understanding
• summary-state memory
• delayed query
• prototype bank