Dynamic Hub-and-Spoke Memory for Streaming Video Understanding

2026-08-31

AI Summary

为解决流视频理解中长时记忆存储与检索问题，提出动态中心辐射记忆框架，通过文本化历史和视觉令牌结合提高问答准确性。

Abstract

Streaming video understanding requires answering questions at arbitrary times over a continuously growing visual stream. The central challenge is to compactly remember long-range history while effectively retrieving question-relevant evidence. We propose Dynamic Hub-and-Spoke Memory (D-HSM), a training-free framework that represents distant history as structured textual memory while preserving the recent frames as visual tokens for fine-grained perception. Specifically, D-HSM turns selected historical video chunks into typed textual observations and stores them in an entity-centered hub-and-spoke memory, with entities as hubs and related evidence as spokes. When answering a question, D-HSM dynamically retrieves a compact question-aware memory subset, expands it through hub-and-spoke links, and combines it with the recent visual window for frozen-VLM answer prediction. Extensive experiments on both streaming and long video benchmarks show that D-HSM consistently and substantially improves VLM backbones and outperforms other state-of-the-art online and offline video understanding baselines.

Problem

• Streaming Video Understanding
• Long-range History
• Question-relevant Evidence

Innovation

• Dynamic Hub-and-Spoke Memory
• streaming video understanding
• entity-centered memory
• question-aware retrieval
• frozen-VLM