Gaussian Mixture Modeling for Event-Aware Visual Allocation in Long Video Understanding

2026-07-14

AI Summary

This work addresses the high computational cost, loss of critical information, and redundancy in long-form video understanding with large vision-language models caused by uniform frame sampling. To this end, it introduces the first training-free, plug-and-play, event-aware visual token allocation framework. The method leverages a Gaussian Mixture Model to infer event-level structures from discretely sampled frames and employs a differentiated multi-resolution strategy: each detected event retains one high-resolution primary keyframe to preserve fine-grained details, complemented by low-resolution secondary keyframes to maintain temporal context. This approach maximizes the utility of a constrained visual token budget. Experiments demonstrate that the proposed method achieves or surpasses state-of-the-art performance on multiple long-video benchmarks while using only approximately half the number of visual tokens.

Abstract

Large Vision-Language Models (LVLMs) face significant challenges in long video understanding due to the excessive computational cost and information loss associated with uniform sampling. Existing keyframe selection methods often treat video frames as atomic entities and allocate visual budgets equally, thereby overlooking high-level semantic structures and introducing substantial redundancy. To address these limitations, we propose GMM-EVA (Gaussian Mixture Modeling for Event-Aware Visual Allocation), which leverages Gaussian Mixture Models to model event-level structure from discrete frame-wise observations. A differentiated allocation strategy is then applied to preserve one primary high-resolution keyframe per event for high-fidelity detail, while utilizing lower-resolution secondary keyframes to maintain temporal context and optimize token budgets. GMM-EVA is a training-free, plug-and-play framework that generalizes robustly across various relevance measures and downstream LVLMs. Extensive experiments on multiple long video benchmarks demonstrate that our method significantly outperforms uniform sampling. Notably, GMM-EVA achieves comparable performance to baseline selection methods while utilizing only approximately half of the visual token budget, highlighting its superior efficiency and effectiveness.

Problem

• long video understanding
• keyframe selection
• visual token budget
• semantic structure
• information redundancy

Innovation

• Gaussian Mixture Model
• Event-Aware Allocation
• Keyframe Selection
• Visual Token Budget
• Long Video Understanding