VC-LLM: Automated Advertisement Video Creation from Raw Footage using Multi-modal LLMs

2025-04-08

AI Summary

To address the low efficiency and limited creative diversity in manually produced short-video advertisements, this paper proposes the first end-to-end framework for automatic advertising video generation. Methodologically, it introduces a dual-path video representation—high-resolution spatial and low-resolution temporal pathways—to enable spatiotemporal disentangled modeling; incorporates a rewrite-based supervision training strategy to mitigate script hallucination; and synergistically integrates multimodal large language models (GPT-4o and a custom fine-tuned LLM) for joint script generation and shot editing. Contributions include: (1) establishing the first benchmark for evaluating advertising video generation; (2) releasing a high-quality pretraining dataset and a human-annotated fine-tuning dataset; and (3) demonstrating experimentally that generated videos achieve human-level visual quality, with the fine-tuned model significantly outperforming the GPT-4o baseline in narrative coherence and logical consistency.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Generation

Application Category

• Search and Retrieval-Augmented AI: Ad search and search for Web retail
• User Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

As short videos have risen in popularity, the role of video content in advertising has become increasingly significant. Typically, advertisers record a large amount of raw footage about the product and then create numerous different short-form advertisement videos based on this raw footage. Creating such videos mainly involves editing raw footage and writing advertisement scripts, which requires a certain level of creative ability. It is usually challenging to create many different video contents for the same product, and manual efficiency is often low. In this paper, we present VC-LLM, a framework powered by Large Language Models for the automatic creation of high-quality short-form advertisement videos. Our approach leverages high-resolution spatial input and low-resolution temporal input to represent video clips more effectively, capturing both fine-grained visual details and broader temporal dynamics. In addition, during training, we incorporate supplementary information generated by rewriting the ground truth text, ensuring that all key output information can be directly traced back to the input, thereby reducing model hallucinations. We also designed a benchmark to evaluate the quality of the created videos. Experiments show that VC-LLM based on GPT-4o can produce videos comparable to those created by humans. Furthermore, we collected numerous high-quality short advertisement videos to create a pre-training dataset and manually cleaned a portion of the data to construct a high-quality fine-tuning dataset. Experiments indicate that, on the benchmark, the VC-LLM based on fine-tuned LLM can produce videos with superior narrative logic compared to those created by the VC-LLM based on GPT-4o.

Problem

• Automates creation of ad videos from raw footage using multi-modal LLMs
• Improves video quality by combining spatial and temporal input representations
• Reduces model hallucinations with traceable input-output training data

Innovation

• Multi-modal LLMs for automatic video creation
• High-resolution spatial and low-resolution temporal inputs
• Supplementary training info to reduce hallucinations