Problem
Research questions and friction points this paper is trying to address.
Instruction-based video editing
Image-to-video transfer
Video latent space
Generative model adaptation
Qwen-Video-Edit: Instruction-Based Video Editing by Repurposing an Image Editing Model
🤖 AI Summary
This study addresses the reliance of instruction-based video editing on costly video generation backbones by proposing a novel adaptation strategy. We rearrange video VAE latencies into virtual image patches, enabling the direct reuse of positional encodings and projection layers from established image editing models. Fine-tuned on the Ditto-1M dataset with Wan2.2 temporal augmentation, this approach achieves high-quality instruction video editing at minimal adaptation cost. Experimental results validate that image editing priors can be efficiently transferred to the video domain, demonstrating significant compatibility between image and video latent spaces. Consequently, this work establishes an efficient paradigm for video editing that circumvents the need for extensive video pre-training, offering a practical alternative to resource-intensive generative frameworks.
📝 Abstract
Instruction-based video editing is commonly built on video-pretrained generative backbones: a video diffusion transformer is adapted, at considerable cost, to condition on a source video and an editing instruction. In this report we explore a different route and show that a strong instruction-based image editing model can edit videos by operating directly on video-VAE latents. Starting from Qwen-Image-Edit, we arrange the latent frames of a Wan~2.1 video VAE as tiles of one large virtual image, reuse the editor's image positional encoding for every tile, and bridge the two latent spaces with a pair of lightweight input/output projections warm-started from the editor's own patchify and unpatchify layers, so that at initialization a (static) video is embedded exactly as an image the model already understands. The whole system is then fine-tuned on the public Ditto-1M editing triplets, and a few denoising steps of Wan~2.2 serve as an optional temporal enhancer. We motivate the design with a chain of zero-training observations: the stock image editor already edits a video presented as a contact sheet; it is indifferent to whether the sheet's tokens come from one joint encode or from per-frame encodes stitched in latent space; and it even edits genuine video latents zero-shot to a clearly recognizable degree, leaving fine-tuning only a fidelity gap to close. Our results suggest that, despite the large investment in training video latent spaces, per-frame video latents remain close enough to the image domain that mature image editing priors transfer with minimal adaptation. Project Page: https://yunpeng1998.github.io/Qwen-Video-Edit-Page; Code: https://github.com/yunpeng1998/Qwen-Video-Edit; Model: https://huggingface.co/yunpeng1998/Qwen-Video-Edit.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Instruction-Based Video Editing
Image-to-Video Transfer
Latent Space Tiling
Lightweight Projection Bridge
Zero-Shot Prior Transfer
🔎 Similar Papers
No similar papers found.
