StreamKV: Streaming Video Question-Answering with Segment-based KV Cache Retrieval and Compression

2025-11-10

AI Summary

Existing Video-LLMs suffer from inefficient KV cache retrieval and compression-induced distortions, leading to suboptimal trade-offs between accuracy and latency in long-video streaming question answering. To address this, we propose StreamKV—a training-free framework featuring a novel semantic-driven dynamic segmentation and layer-adaptive KV caching co-optimization mechanism. Specifically, it generates segment-level summary vectors via semantic video segmentation for precise KV retrieval, and introduces guided semantic-aware compression to perform cross-layer adaptive cache pruning within a unified module. Crucially, StreamKV requires no fine-tuning, enhancing both efficiency and robustness for long-video understanding. On the StreamingVQA benchmark, StreamKV achieves an 8.2% absolute accuracy gain over state-of-the-art online Video-LLMs, while reducing memory footprint by 37% and inference latency by 29%.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Video Understanding & Activity Analysis
• Search and Optimization: Learning to Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Video Large Language Models (Video-LLMs) have demonstrated significant potential in the areas of video captioning, search, and summarization. However, current Video-LLMs still face challenges with long real-world videos. Recent methods have introduced a retrieval mechanism that retrieves query-relevant KV caches for question answering, enhancing the efficiency and accuracy of long real-world videos. However, the compression and retrieval of KV caches are still not fully explored. In this paper, we propose extbf{StreamKV}, a training-free framework that seamlessly equips Video-LLMs with advanced KV cache retrieval and compression. Compared to previous methods that used uniform partitioning, StreamKV dynamically partitions video streams into semantic segments, which better preserves semantic information. For KV cache retrieval, StreamKV calculates a summary vector for each segment to retain segment-level information essential for retrieval. For KV cache compression, StreamKV introduces a guidance prompt designed to capture the key semantic elements within each segment, ensuring only the most informative KV caches are retained for answering questions. Moreover, StreamKV unifies KV cache retrieval and compression within a single module, performing both in a layer-adaptive manner, thereby further improving the effectiveness of streaming video question answering. Extensive experiments on public StreamingVQA benchmarks demonstrate that StreamKV significantly outperforms existing Online Video-LLMs, achieving superior accuracy while substantially improving both memory efficiency and computational latency. The code has been released at https://github.com/sou1p0wer/StreamKV.

Problem

• StreamKV addresses efficient KV cache retrieval for long video question-answering
• It dynamically segments videos to preserve semantic information better
• The framework unifies KV cache compression and retrieval for improved performance

Innovation

• Dynamically partitions video streams into semantic segments
• Calculates summary vectors for segment-level retrieval
• Unifies retrieval and compression with guidance prompts