Question-guided Visual Compression with Memory Feedback for Long-Term Video Understanding

2026-03-16

AI Summary

Existing approaches to long video understanding exhibit limited performance on tasks requiring holistic event modeling—such as temporal ordering—due to their frame-level independent compression and the absence of an interaction mechanism between perception and memory. To address this, this work proposes QViC-MF, a novel framework that introduces a memory feedback mechanism into the visual compression process for the first time. By leveraging question-guided multimodal selective attention (QMSa), QViC-MF establishes iterative interactions between the current video segment and relevant historical frames stored in memory, enabling task-driven dynamic context awareness. The proposed method achieves significant improvements over state-of-the-art approaches, with absolute gains of 6.1%, 8.3%, 18.3%, and 3.7% on MLVU, LVBench, VNBench Long, and VideoMME Long, respectively.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Machine Learning: Multimodal Learning
• Data Mining & Knowledge Management: Mining of Visual, Multimedia & Multimodal Data

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

In the context of long-term video understanding with large multimodal models, many frameworks have been proposed. Although transformer-based visual compressors and memory-augmented approaches are often used to process long videos, they usually compress each frame independently and therefore fail to achieve strong performance on tasks that require understanding complete events, such as temporal ordering tasks in MLVU and VNBench. This motivates us to rethink the conventional one-way scheme from perception to memory, and instead establish a feedbackdriven process in which past visual contexts stored in the context memory can benefit ongoing perception. To this end, we propose Question-guided Visual Compression with Memory Feedback (QViC-MF), a framework for long-term video understanding. At its core is a Question-guided Multimodal Selective Attention (QMSA), which learns to preserve visual information related to the given question from both the current clip and the past related frames from the memory. The compressor and memory feedback work iteratively for each clip of the entire video. This simple yet effective design yields large performance gains on longterm video understanding tasks. Extensive experiments show that our method achieves significant improvement over current state-of-the-art methods by 6.1% on MLVU test, 8.3% on LVBench, 18.3% on VNBench Long, and 3.7% on VideoMME Long. The code will be released publicly.

Problem

• long-term video understanding
• visual compression
• temporal ordering
• memory feedback
• multimodal models

Innovation

• Memory Feedback
• Question-guided Attention
• Visual Compression
• Long-term Video Understanding
• Multimodal Selective Attention