RACCooN: A Versatile Instructional Video Editing Framework with Auto-Generated Narratives

2024-05-28

AI Summary

Existing video editing methods rely on manually crafted text prompts, limiting flexibility and scalability. This paper proposes V2P-P2V, an end-to-end video editing framework: first, it automatically encodes input videos into structured natural language descriptions via unsupervised multi-granularity spatiotemporal pooling (Video-to-Prompt, V2P); second, users freely edit these descriptions, which then condition a diffusion model to generate edited videos (Prompt-to-Video, P2V), enabling fine-grained operations—including object removal, insertion, replacement, and “imagining novel objects.” Key contributions include: (i) the first annotation-free mechanism for video description generation; (ii) LLM-enhanced automatic narrative instruction synthesis; and (iii) explicit editing plan generation. Experiments demonstrate significant improvements over baselines in both video captioning and controllable editing tasks, and show that V2P-P2V can be seamlessly integrated to enhance the editing capability of state-of-the-art video generation models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Recent video generative models primarily rely on carefully written text prompts for specific tasks, like inpainting or style editing. They require labor-intensive textual descriptions for input videos, hindering their flexibility to adapt personal/raw videos to user specifications. This paper proposes RACCooN, a versatile and user-friendly video-to-paragraph-to-video generative framework that supports multiple video editing capabilities such as removal, addition, and modification, through a unified pipeline. RACCooN consists of two principal stages: Video-to-Paragraph (V2P) and Paragraph-to-Video (P2V). In the V2P stage, we automatically describe video scenes in well-structured natural language, capturing both the holistic context and focused object details. Subsequently, in the P2V stage, users can optionally refine these descriptions to guide the video diffusion model, enabling various modifications to the input video, such as removing, changing subjects, and/or adding new objects. The proposed approach stands out from other methods through several significant contributions: (1) RACCooN suggests a multi-granular spatiotemporal pooling strategy to generate well-structured video descriptions, capturing both the broad context and object details without requiring complex human annotations, simplifying precise video content editing based on text for users. (2) Our video generative model incorporates auto-generated narratives or instructions to enhance the quality and accuracy of the generated content. (3) RACCooN also plans to imagine new objects in a given video, so users simply prompt the model to receive a detailed video editing plan for complex video editing. The proposed framework demonstrates impressive versatile capabilities in video-to-paragraph generation, video content editing, and can be incorporated into other SoTA video generative models for further enhancement.

Problem

• Automates video description generation to enable text-guided editing without manual annotations
• Supports versatile video modifications through unified paragraph-to-video generation pipeline
• Enhances video editing accessibility using auto-generated narratives and user refinements

Innovation

• Automatically generates structured video descriptions without annotations
• Uses auto-generated narratives to enhance video editing accuracy
• Enables complex video modifications through simple user prompts