Parameter-free Video Segmentation for Vision and Language Understanding

2025-03-03

AI Summary

Existing video segmentation methods for long videos require predefined priors—such as thresholds, target segment counts, or length constraints—leading to inaccurate boundary localization. To address this, we propose a fully parameter-free video segmentation framework that automatically identifies semantically coherent segment boundaries without any human-specified hyperparameters. Our approach innovatively integrates the Minimum Description Length (MDL) principle with dynamic programming, enabling principled, data-driven boundary detection. It employs frame-level multimodal feature modeling and an optimized boundary search strategy to closely approximate the intrinsic structural semantics of real-world scenes. Evaluated on long-video summarization and retrieval-augmented video question answering, our method consistently outperforms state-of-the-art segmentation baselines. Downstream task performance improves significantly, demonstrating strong generalizability and practical utility across diverse video understanding applications.

Technology Category

• Computer Vision: Segmentation
• Machine Learning: Multimodal Learning
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations

Abstract

The proliferation of creative video content has driven demand for adapting language models to handle video input and enable multimodal understanding. However, end-to-end models struggle to process long videos due to their size and complexity. An effective alternative is to divide them into smaller chunks to be processed separately, and this motivates a method for choosing where the chunk boundaries should be. In this paper, we propose an algorithm for segmenting videos into contiguous chunks, based on the minimum description length principle, coupled with a dynamic programming search. The algorithm is entirely parameter-free, given feature vectors, not requiring a set threshold or the number or size of chunks to be specified. We show empirically that the breakpoints it produces more accurately approximate scene boundaries in long videos, compared with existing methods for scene detection, even when such methods have access to the true number of scenes. We then showcase this algorithm in two tasks: long video summarization, and retrieval-augmented video question answering. In both cases, scene breaks produced by our algorithm lead to better downstream performance than existing methods for video segmentation.

Problem

• Adapt language models for video input and multimodal understanding.
• Segment long videos into chunks without predefined parameters.
• Improve video summarization and question answering with accurate scene breaks.

Innovation

• Parameter-free video segmentation algorithm
• Uses minimum description length principle
• Dynamic programming for scene boundary detection