Goku: A Million-Scale Universal Dataset and Benchmark for Instruction-Based Video Editing

2026-06-29

AI Summary

This work addresses the limitation of existing video editing datasets, which predominantly focus on single-task appearance editing and lack support for structured operations such as subject motion control. To overcome this, the authors introduce the Goku project, comprising a large-scale dataset of 2 million high-quality instruction-aligned samples, curated through controllable subproblem decomposition and progressive filtering to ensure reliability. They propose Goku-Edit, a novel model featuring a decoupled dual-branch architecture that separates structural control from appearance rendering, leveraging a multimodal large language model as the text encoder. Additionally, they establish Goku-Bench, the first benchmark tailored for complex video editing instructions, incorporating seven specialized metrics and 1,000 human-verified test cases. Experiments demonstrate that Goku-Edit achieves up to an 8% improvement in instruction-following capability over current open-source models.

Abstract

Existing instruction-based video editing datasets commonly focus on single-task appearance editing, failing to meet the complex creative demands of real-world scenarios. To bridge this gap, we present Goku, a large-scale dataset featuring 2 million high-quality, instruction-aligned video editing pairs, which is the first to extend task boundaries from basic appearance editing to multi-task and structural manipulations(e.g., precise control of subject movement). To tackle the data synthesis challenges inherent in these complex tasks, we design an efficient data synthesis pipeline that decomposes complex edits into controllable sub-problems and introduce a progressive filtering system for data reliability throughout the whole process. Furthermore, we explore the optimal network structures on Goku, and propose Goku-Edit. To deeply comprehend complex editing instructions, Goku-Edit leverages an MLLM as its text encoder and adopts a decoupled dual-branch design: a dedicated mask branch handles structural control, freeing the main branch for appearance rendering. A comprehensive video editing benchmark, Goku-Bench, is also proposed with 1,000 human-verified test cases and 7 novel editing-specific metrics. Evaluated on Goku-Bench, Goku-Edit obtains up to +8% improvement on other open-source models in terms of instruction following.

Problem

• instruction-based video editing
• multi-task editing
• structural manipulation
• video editing dataset
• complex editing instructions

Innovation

• instruction-based video editing
• multi-task video manipulation
• data synthesis pipeline
• decoupled dual-branch architecture
• video editing benchmark