Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding

2026-06-24

AI Summary

Existing streaming video understanding poses significant challenges for multimodal large language models due to the unbounded frame count, unpredictable future content, and dynamically evolving instructions. This work proposes CausalMem, the first training-free dynamic fixed-budget memory mechanism that online estimates semantic bases to assess visual token redundancy and continuously updates a memory-constrained visual memory bank, emulating the human brain’s capacity for information compression and retention. CausalMem is compatible with mainstream architectures such as LLaVA-OneVision and Qwen2.5-VL, achieving average accuracy gains of 3.2% and 3.0% on streaming and offline video understanding benchmarks, respectively. With only 82MB of storage, it enables over 20× compression of visual tokens while preserving high-fidelity memory for hour-long videos within a 12k token budget.

Abstract

Currently, streaming video understanding is still a daunting task for existing \emph{multimodal large language models} (MLLMs). Its difficulties not only lie in handling the ever-increasing video frames, but also in the unpredictability of future video content and input instructions. In this paper, we study this task from the perspective of constructing a dynamic but fixed-budget memory bank, and propose a novel and training-free approach termed \emph{\textbf{CausalMem}}. CausalMem is dedicated to constructing a dynamic visual memory update mechanism, thereby maximizing the amount of information in streaming video within a limited memory space, much like the human brain. In practice, CausalMem estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis, which models the principal semantics of the observed video stream. To validate CausalMem, we apply it to two representative MLLMs, namely LLaVA-OneVision and Qwen2.5-VL respectively, and conduct extensive experiments on both streaming and offline video understanding benchmarks. The experimental results not only show the great advantages than existing methods under both streaming and offline settings, \emph{e.g.}, $+3.2\%$ and $+3.0\%$ average accuracy gains respectively, but also witness the superior semantic preservation for streaming videos, \emph{e.g.}, using 12$k$ token budgets to memorize hour-long streaming videos, which achieves more than \textbf{20$\times$} visual token compression ratio and only occupies about \textbf{82 MB} storage. \textbf{Our code} is given in \href{https://github.com/hktk07/CausalMem}{CausalMem}.

Problem

• streaming video understanding
• multimodal large language models
• fixed-budget memory
• visual memory
• token compression

Innovation

• CausalMem
• fixed-budget memory bank
• streaming video understanding
• visual token compression
• online semantic basis