$infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation

2025-01-31

AI Summary

Existing video-language models suffer from fixed context windows and sparse frame sampling, hindering effective modeling of continuous temporal structure in long videos and leading to critical information loss. To address this, we propose a training-free, length-unbounded Long-Term Memory (LTM) mechanism grounded in continuous-time modeling. Our approach features: (1) “sticky” memory evolution via continuous-time attention, enabling adaptive memory updating and retention; (2) enhanced spatiotemporal representation through video Q-former integration and dynamic-granularity memory allocation; and (3) zero-shot transferability to mainstream architectures—including Video-LLaMA and VideoChat2—without architectural modification or fine-tuning. Evaluated on video question answering, our method achieves significant performance gains over baselines while maintaining computational efficiency. It demonstrates scalable, training-free long-video understanding with minimal overhead, advancing practical deployment of video-language models for extended temporal reasoning.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Current video-language models struggle with long-video understanding due to limited context lengths and reliance on sparse frame subsampling, often leading to information loss. This paper introduces $infty$-Video, which can process arbitrarily long videos through a continuous-time long-term memory (LTM) consolidation mechanism. Our framework augments video Q-formers by allowing them to process unbounded video contexts efficiently and without requiring additional training. Through continuous attention, our approach dynamically allocates higher granularity to the most relevant video segments, forming"sticky"memories that evolve over time. Experiments with Video-LLaMA and VideoChat2 demonstrate improved performance in video question-answering tasks, showcasing the potential of continuous-time LTM mechanisms to enable scalable and training-free comprehension of long videos.

Problem

• Long Video Analysis
• Memory Updating
• Video-Language Models

Innovation

• Long-Term Memory
• Arbitrary Length Videos
• Video Understanding