FOLIO: Focused Semantic Memory for Streaming Video Understanding

2026-07-14

AI Summary

This work addresses the challenge of online streaming video understanding, which requires efficiently compressing and preserving historical information under unknown future inputs and query conditions while avoiding memory explosion. The authors propose a training-free, focus-driven semantic memory system that leverages a dynamic focusing mechanism to guide clip-level memory updates. It integrates a short-term visual buffer with an entity-centric long-term semantic memory, caching key visual evidence only for salient entities while compactly storing non-essential context. This design enables lightweight hybrid retrieval through structured memory organization and semantic query expansion. Evaluated on OVO-Bench, the method achieves 82.0% and 69.1% accuracy in perception and retrospective tasks, respectively, and attains a 74.5% overall accuracy on StreamingBench, significantly reducing the computational cost of maintaining streaming memory.

Abstract

In online streaming video understanding, a video stream continues to arrive and queries may be issued at any time. Because streaming frames grow without bound, the system must continuously compress and retain information from the observed video prefix while future frames and future queries remain unknown. The core challenge is deciding what information to retain and how to organize the maintained history: as this history grows with the stream, memory cost increases and many redundant visual details are retained, whereas later queries often depend on specific entities, actions, and their temporal changes. To address this challenge, we introduce FOLIO, a training-free focused semantic memory system that records important parts of the stream in higher detail while keeping surrounding context compact. As the stream arrives, FOLIO updates memory at the segment level, guided by a dynamic focus state, combining a short-term visual buffer with a long-term semantic memory organized around observed entities and linked to a visual-evidence cache. At query time, lightweight hybrid retrieval combines direct matching over the structured memory with semantic query expansion. FOLIO achieves state-of-the-art performance, reaching 82.0/69.1 Perception/Backward accuracy on OVO-Bench with Qwen3-VL-8B and 74.5 overall accuracy on StreamingBench, while substantially reducing the cost of maintaining streaming memory by reserving detailed records for focused entities and storing surrounding context compactly.

Problem

• streaming video understanding
• semantic memory
• memory compression
• online video processing
• temporal reasoning

Innovation

• focused semantic memory
• streaming video understanding
• dynamic focus state
• hybrid retrieval
• training-free