LiveStarPro: Proactive Streaming Video Understanding with Hierarchical Memory for Long-Horizon Streams

2026-06-16

AI Summary

This work addresses the limitations of existing video large language models in processing continuous video streams, where high latency and severe memory decay arise from their inability to respond in real time, autonomously determine response timing, and maintain long-term contextual memory. To overcome these challenges, the authors propose an active comprehension system for long-horizon video streams, integrating three core techniques: Streaming Verification Decoding (SVeD) enables precise response timing without silent tokens; Streaming Causal Attention Masks (SCAM) facilitate variable-length stream-aligned training; and Tree-Structured Hierarchical Memory (TSHM) leverages streaming key-value caching to efficiently organize and retrieve nearly unbounded historical information. Evaluated on the newly introduced OmniStarPro benchmark, the system achieves a 28.9% improvement in semantic correctness, an 18.2% reduction in response timing error, and a 1.58× speedup in inference.

Abstract

Despite the remarkable progress of Video Large Language Models (Video-LLMs), current online architectures still struggle to simultaneously process continuous video streams, decide autonomously when to respond, and preserve long-horizon contextual memory. These obstacles undermine real-time responsiveness and cause severe forgetting throughout prolonged interactions. In this work, we introduce LiveStarPro, a live streaming assistant that is designed for proactive video understanding over long-horizon streams. The design of LiveStarPro rests on three complementary components. The first component is Streaming Verification Decoding (SVeD), an inference framework that identifies the appropriate response timing through single-pass perplexity verification, thereby eliminating the dependency on explicit silence tokens. The second component is Streaming Causal Attention Masks (SCAM), a training strategy that enforces incremental video-language alignment over variable-length streams. The third component is Tree-Structured Hierarchical Memory (TSHM), a recursive memory architecture that organizes evicted historical information into event chains and consequently enables efficient retrieval from effectively unbounded video streams. To facilitate a comprehensive evaluation under realistic online conditions, we further present OmniStarPro, a large-scale benchmark that spans 15 diverse real-world scenarios and that extends to hour-scale streams for the assessment of long-term recall. Extensive experiments demonstrate that LiveStarPro consistently surpasses existing methods, attaining a 28.9% improvement in semantic correctness and an 18.2% reduction in timing error, while its streaming key-value cache further yields a 1.58x inference speedup over the same model without caching. The model and the code are publicly available at https://github.com/sotayang/LiveStarPro.

Problem

• Video-LLMs
• long-horizon streams
• contextual memory
• real-time responsiveness
• streaming video understanding

Innovation

• Streaming Verification Decoding
• Streaming Causal Attention Masks
• Tree-Structured Hierarchical Memory
• Video Large Language Models
• Long-Horizon Video Understanding