StreamChat: Chatting with Streaming Video

2024-12-11

arXiv.org

AI Summary

Existing large multimodal models (LMMs) for streaming video interaction rely solely on static visual inputs captured at the query timestamp, rendering them incapable of perceiving subsequent frame dynamics and resulting in delayed responses. StreamChat addresses this by introducing a dynamic visual context updating mechanism—per-decoding-step—enabling fine-grained coordination between large vision-language models and real-time video streams. Its core contributions are: (1) the first streaming visual token incremental injection scheme coupled with a cross-modal dynamic attention architecture; (2) the construction of the first densely annotated streaming instruction-tuning dataset; and (3) parallel 3D-RoPE encoding, which explicitly models temporal relationships between visual and textual sequences. Experiments demonstrate that StreamChat maintains competitive performance on standard image/video benchmarks while significantly outperforming existing video LMMs on streaming interaction tasks, with controllable inference latency and markedly improved response timeliness.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper presents StreamChat, a novel approach that enhances the interaction capabilities of Large Multimodal Models (LMMs) with streaming video content. In streaming interaction scenarios, existing methods rely solely on visual information available at the moment a question is posed, resulting in significant delays as the model remains unaware of subsequent changes in the streaming video. StreamChat addresses this limitation by innovatively updating the visual context at each decoding step, ensuring that the model utilizes up-to-date video content throughout the decoding process. Additionally, we introduce a flexible and efficient crossattention-based architecture to process dynamic streaming inputs while maintaining inference efficiency for streaming interactions. Furthermore, we construct a new dense instruction dataset to facilitate the training of streaming interaction models, complemented by a parallel 3D-RoPE mechanism that encodes the relative temporal information of visual and text tokens. Experimental results demonstrate that StreamChat achieves competitive performance on established image and video benchmarks and exhibits superior capabilities in streaming interaction scenarios compared to state-of-the-art video LMM.

Problem

• Enhancing LMMs' interaction with streaming video content
• Reducing delays by updating visual context dynamically
• Improving efficiency in processing dynamic streaming inputs

Innovation

• Updates visual context at each decoding step
• Uses crossattention-based architecture for dynamic inputs
• Introduces 3D-RoPE for temporal information encoding