Aurora: Unified Video Editing with a Tool-Using Agent

2026-05-18

AI Summary

Existing unified video editing approaches rely heavily on users providing precise textual instructions, reference images, and spatial localization, making them ill-suited for handling ambiguous or incomplete editing requests in real-world scenarios. This work proposes an agent-driven video editing framework that introduces, for the first time, a vision-language model (VLM) agent equipped with tool-calling capabilities to automatically interpret vague instructions, select appropriate reference images, and generate structured editing plans. The agent collaborates with a unified video diffusion Transformer to execute diverse editing tasks. By integrating supervised training, preference optimization, and tool-augmented reasoning, the method significantly outperforms instruction-only baselines on AgentEdit-Bench and two established benchmarks. Moreover, the VLM agent demonstrates strong generalization, effectively enhancing other frozen video editing models without retraining.

Abstract

Recent video editing models have converged on a unified conditioning design: a single diffusion transformer jointly consumes text, source video, and reference images, and one set of weights covers replacement, removal, style transfer, and reference-driven insertion. The design is flexible, but it assumes that the user already provides model-ready text, reference images, and spatial grounding for local edits, which real requests often omit. We present Aurora, an agentic video editing framework that pairs a tool-augmented vision-language model (VLM) agent with a unified video diffusion transformer. The VLM agent maps a raw user request to a structured edit plan aligned with the transformer's conditioning channels, thereby resolving textual and visual underspecification before generation. We train the VLM agent with supervised data for complete edit planning and reference-image selection, together with preference pairs for robust tool use and instruction refinement. We introduce AgentEdit-Bench to evaluate agent-enhanced video editing under textual and visual underspecification. Experiments on AgentEdit-Bench and two existing video editing benchmarks show that Aurora improves over instruction-only baselines and that the VLM agent transfers to compatible frozen video editing models. Project page: https://yeates.github.io/Aurora-Page

Problem

• video editing
• textual underspecification
• visual underspecification
• user request interpretation
• reference image selection

Innovation

• tool-using agent
• unified video editing
• vision-language model
• edit planning
• diffusion transformer