TimeChat-Online: 80% Visual Tokens are Naturally Redundant in Streaming Videos

2025-04-24

AI Summary

To address the high computational overhead and latency of VideoLLMs in real-time video stream processing caused by frame-level redundancy, this paper introduces the first online-interactive streaming video large language model. Our core innovation is the Differential Token Dropping (DTD) module, inspired by human change blindness, which identifies and exploits the inherent redundancy—82.8% of visual tokens—in streaming video without language guidance, enabling proactive, language-free response. Integrated with a lightweight streaming architecture and trained on our newly curated TimeChat-Online-139K dataset—covering multi-scenario interactive tasks including retrospective reasoning, perception, and prediction—the model achieves an 82.8% token compression rate on StreamingBench while retaining 98% of baseline performance, significantly outperforming prior methods. It also demonstrates competitive results on OvOBench, Video-MME, and MLVU benchmarks.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The rapid growth of online video platforms, particularly live streaming services, has created an urgent need for real-time video understanding systems. These systems must process continuous video streams and respond to user queries instantaneously, presenting unique challenges for current Video Large Language Models (VideoLLMs). While existing VideoLLMs excel at processing complete videos, they face significant limitations in streaming scenarios due to their inability to handle dense, redundant frames efficiently. We introduce TimeChat-Online, a novel online VideoLLM that revolutionizes real-time video interaction. At its core lies our innovative Differential Token Drop (DTD) module, which addresses the fundamental challenge of visual redundancy in streaming videos. Drawing inspiration from human visual perception's Change Blindness phenomenon, DTD preserves meaningful temporal changes while filtering out static, redundant content between frames. Remarkably, our experiments demonstrate that DTD achieves an 82.8% reduction in video tokens while maintaining 98% performance on StreamingBench, revealing that over 80% of visual content in streaming videos is naturally redundant without requiring language guidance. To enable seamless real-time interaction, we present TimeChat-Online-139K, a comprehensive streaming video dataset featuring diverse interaction patterns including backward-tracing, current-perception, and future-responding scenarios. TimeChat-Online's unique Proactive Response capability, naturally achieved through continuous monitoring of video scene transitions via DTD, sets it apart from conventional approaches. Our extensive evaluation demonstrates TimeChat-Online's superior performance on streaming benchmarks (StreamingBench and OvOBench) and maintaining competitive results on long-form video tasks such as Video-MME and MLVU.

Problem

• Handles real-time video understanding for streaming services efficiently
• Reduces visual redundancy in streaming videos without performance loss
• Enables seamless interaction with proactive response to scene changes

Innovation

• Differential Token Drop reduces visual redundancy
• Proactive Response via continuous scene monitoring
• TimeChat-Online-139K dataset enables diverse interactions