Problem
Research questions and friction points this paper is trying to address.
advertisement video editing
multimodal integration
production efficiency
scalable content creation
end-to-end framework
AutoCut: End-to-end advertisement video editing based on multimodal discretization and controllable generation
🤖 AI Summary
Current workflows for producing short advertising videos suffer from fragmented pipelines and isolated modalities, resulting in high costs and low efficiency. This work proposes the first end-to-end multimodal framework for ad video editing, which unifies video, audio, and text representations within a shared discrete token space. By leveraging a multimodal large language model, the framework enables coordinated and controllable generation across key tasks—including scriptwriting, shot selection, and music scoring. The approach integrates specialized encoders, residual vector quantization, multimodal alignment, and supervised fine-tuning. Evaluated on real-world advertising datasets, it significantly reduces production costs and iteration time while enhancing content consistency and editing controllability.
📝 Abstract
Short-form videos have become a primary medium for digital advertising, requiring scalable and efficient content creation. However, current workflows and AI tools remain disjoint and modality-specific, leading to high production costs and low overall efficiency. To address this issue, we propose AutoCut, an end-to-end advertisement video editing framework based on multimodal discretization and controllable editing. AutoCut employs dedicated encoders to extract video and audio features, then applies residual vector quantization to discretize them into unified tokens aligned with textual representations, constructing a shared video-audio-text token space. Built upon a foundation model, we further develop a multimodal large language model for video editing through combined multimodal alignment and supervised fine-tuning, supporting tasks covering video selection and ordering, script generation, and background music selection within a unified editing framework. Finally, a complete production pipeline converts the predicted token sequences into deployable long video outputs. Experiments on real-world advertisement datasets show that AutoCut reduces production cost and iteration time while substantially improving consistency and controllability, paving the way for scalable video creation.
Innovation
Methods, ideas, or system contributions that make the work stand out.
multimodal discretization
controllable generation
end-to-end video editing
multimodal large language model
shared token space
🔎 Similar Papers
No similar papers found.
