From Long Videos to Engaging Clips: A Human-Inspired Video Editing Framework with Multimodal Narrative Understanding

2025-07-03

AI Summary

Existing automatic video editing methods over-rely on ASR transcripts while neglecting visual context, resulting in narratively incoherent outputs. To address this, we propose HIVE, the first end-to-end video editing framework integrating multimodal narrative understanding. HIVE decomposes editing into three stages—highlight detection, head/tail selection, and redundancy removal—guided by character extraction, dialogue analysis, and narrative summarization. It synergistically fuses multimodal large language models, scene-level segmentation, ASR transcripts, and visual context modeling to emulate human editorial reasoning. Evaluated on our newly constructed dataset DramaAD, HIVE achieves significant improvements in narrative coherence and readability for both general and advertisement-oriented short-video generation. Quantitatively, it substantially narrows the quality gap between automated editing and professional human editing, demonstrating superior alignment with human perceptual and narrative expectations.

Technology Category

• Computer Vision: Multi-modal Vision
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

The rapid growth of online video content, especially on short video platforms, has created a growing demand for efficient video editing techniques that can condense long-form videos into concise and engaging clips. Existing automatic editing methods predominantly rely on textual cues from ASR transcripts and end-to-end segment selection, often neglecting the rich visual context and leading to incoherent outputs. In this paper, we propose a human-inspired automatic video editing framework (HIVE) that leverages multimodal narrative understanding to address these limitations. Our approach incorporates character extraction, dialogue analysis, and narrative summarization through multimodal large language models, enabling a holistic understanding of the video content. To further enhance coherence, we apply scene-level segmentation and decompose the editing process into three subtasks: highlight detection, opening/ending selection, and pruning of irrelevant content. To facilitate research in this area, we introduce DramaAD, a novel benchmark dataset comprising over 800 short drama episodes and 500 professionally edited advertisement clips. Experimental results demonstrate that our framework consistently outperforms existing baselines across both general and advertisement-oriented editing tasks, significantly narrowing the quality gap between automatic and human-edited videos.

Problem

• Condense long videos into engaging short clips
• Improve coherence using multimodal narrative understanding
• Bridge quality gap between automatic and human editing

Innovation

• Multimodal narrative understanding for video editing
• Character extraction and dialogue analysis integration
• Scene-level segmentation for enhanced coherence