Region-Constraint In-Context Generation for Instructional Video Editing

2025-12-19

AI Summary

To address inaccurate region localization, inter-frame drift, and token interference in instruction-driven video editing, this paper proposes a region-constrained contextual generation paradigm. Methodologically, it introduces the first video-level region constraint modeling framework; designs a dual regularization mechanism comprising latent-space difference regularization and attention masking suppression; and incorporates width-wise source-target video concatenation with one-step backward denoising optimization. Contributions include: (1) constructing ReCo-Data—the first large-scale, high-quality dataset of 500K instruction-video pairs; and (2) achieving state-of-the-art performance across four benchmark tasks, with significant improvements in editing region accuracy, inter-frame consistency, and effective suppression of irrelevant region generation and cross-frame interference.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Natural Language Processing: Generation
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Data and user privacy-enhancing technologies for the Web
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

The In-context generation paradigm recently has demonstrated strong power in instructional image editing with both data efficiency and synthesis quality. Nevertheless, shaping such in-context learning for instruction-based video editing is not trivial. Without specifying editing regions, the results can suffer from the problem of inaccurate editing regions and the token interference between editing and non-editing areas during denoising. To address these, we present ReCo, a new instructional video editing paradigm that novelly delves into constraint modeling between editing and non-editing regions during in-context generation. Technically, ReCo width-wise concatenates source and target video for joint denoising. To calibrate video diffusion learning, ReCo capitalizes on two regularization terms, i.e., latent and attention regularization, conducting on one-step backward denoised latents and attention maps, respectively. The former increases the latent discrepancy of the editing region between source and target videos while reducing that of non-editing areas, emphasizing the modification on editing area and alleviating outside unexpected content generation. The latter suppresses the attention of tokens in the editing region to the tokens in counterpart of the source video, thereby mitigating their interference during novel object generation in target video. Furthermore, we propose a large-scale, high-quality video editing dataset, i.e., ReCo-Data, comprising 500K instruction-video pairs to benefit model training. Extensive experiments conducted on four major instruction-based video editing tasks demonstrate the superiority of our proposal.

Problem

• Addresses inaccurate editing regions in instructional video editing
• Mitigates token interference between editing and non-editing areas
• Enhances constraint modeling for precise video content generation

Innovation

• Region-constraint in-context generation for video editing
• Latent and attention regularization for denoising calibration
• Large-scale dataset with 500K instruction-video pairs