Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge

2025-01-23

AI Summary

Existing video understanding models struggle to simultaneously support long-video modeling, multi-turn dialogue consistency, and real-time response. This paper proposes a fine-tuning-free streaming video-language joint understanding framework, introducing a novel hierarchical memory mechanism and parallel scheduling strategy to enable efficient long-video compression, low-latency inference, and cross-turn contextual coherence modeling. Built upon the Video-LLM architecture, our approach integrates hierarchical feature memory, streaming feature compression, dynamic system scheduling, and a knowledge-enhanced dialogue engine. Evaluated on our newly constructed StreamBench benchmark and multiple public benchmarks, it significantly outperforms state-of-the-art methods: achieving a 12.6% accuracy gain, reducing average response latency by 47%, and—critically—enabling the first high-accuracy, low-latency, and sustainable interactive long-video understanding capability.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. $StreamChat$ leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. Extensive evaluations on StreamBench and other public benchmarks demonstrate that StreamChat significantly outperforms existing state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available at StreamChat: https://github.com/hmxiong/StreamChat.

Problem

• Video Understanding
• Long Video Processing
• Adaptive Real-Scene Changes

Innovation

• Real-time Video Processing
• Memory Mechanism
• Parallel Processing