ViCoStream: Streaming VideoLLMs Can Run Beyond 100 FPS with Stage-Wise Coordinated Inference

2026-06-18

AI Summary

This work addresses the challenge of achieving both high throughput and low latency in real-time streaming video large language model (VideoLLM) inference, an area lacking systematic analysis of sustained system-level performance. It introduces ViCoStream, the first framework to model streaming VideoLLMs as a coordinated pipeline encompassing visual preprocessing, encoding, token pruning, and LLM inference. ViCoStream jointly optimizes computation and memory overhead through chunked execution, CUDA stream parallelism, dynamic visual token control, bounded visual attention, and query-side retrieval. Experiments on a single A100 GPU demonstrate that Qwen2.5-VL-3B/7B-Instruct achieves 134 FPS throughput with under 50 ms time-to-first-token latency while maintaining accuracy close to full-history baselines. The study further reveals how chunk size, token retention ratio, attention locality, and retrieval scope govern the trade-off between throughput and accuracy.

Abstract

Streaming VideoLLMs must continuously process incoming video while maintaining low query latency, making both video-ingestion throughput and query-time responsiveness critical for real-time deployment. Existing methods largely focus on accelerating individual modules, such as visual encoding, token pruning, or KV-cache compression, but provide limited insight into whether the resulting system can sustain real-time streaming performance. We formulate streaming VideoLLM inference as a coordinated pipeline spanning visual preprocessing, visual encoding, token dropping, and LLM prefilling/decoding. Building on this formulation, we propose ViCoStream (Video Coordinated Streaming), a stage-wise coordinated streaming framework that combines chunk-wise execution, CUDA-stream overlap, visual token control, bounded visual attention, and query-side retrieval to bound per-chunk computation and memory costs. We further provide a systematic study of bottleneck migration, revealing how chunk size, token retention, attention locality, and retrieval scope shape the throughput-accuracy trade-off. Experiments with Qwen2.5-VL-3B/7B-Instruct across multiple streaming benchmarks show that ViCoStream achieves 134 FPS video throughput and less than 50 ms TTFT on a single A100 GPU while maintaining accuracy close to full-history baselines.

Problem

• Streaming VideoLLMs
• real-time inference
• video throughput
• query latency
• real-time deployment

Innovation

• Streaming VideoLLMs
• Stage-wise Coordinated Inference
• Chunk-wise Execution
• Bounded Visual Attention
• CUDA-stream Overlap