From Plans to Pixels: Learning to Plan and Orchestrate for Open-Ended Image Editing

2026-05-14

AI Summary

This work addresses the challenge that existing image editing models struggle with abstract, multi-step natural language instructions—such as “make the advertisement more vegan-friendly.” The authors propose an end-to-end learnable long-horizon editing framework that decomposes complex instructions into structured atomic tasks via a planner, while a coordinator dynamically selects appropriate editing tools and target regions. A vision-language critic provides outcome-oriented reward signals to guide the process. Notably, this approach tightly couples task planning with reward-driven execution and iteratively refines the planner using successful trajectories, eliminating reliance on handcrafted rules or behavioral cloning from expert demonstrations. Experiments demonstrate that the framework produces significantly more coherent and reliable edits under complex, open-ended instructions compared to single-step models and rule-based multi-step baselines.

Abstract

Modern image editing models produce realistic results but struggle with abstract, multi step instructions (e.g., ``make this advertisement more vegetarian-friendly''). Prior agent based methods decompose such tasks but rely on handcrafted pipelines or teacher imitation, limiting flexibility and decoupling learning from actual editing outcomes. We propose an experiential framework for long-horizon image editing, where a planner generates structured atomic decompositions and an orchestrator selects tools and regions to execute each step. A vision language judge provides outcome-based rewards for instruction adherence and visual quality. The orchestrator is trained to maximize these rewards, and successful trajectories are used to refine the planner. By tightly coupling planning with reward driven execution, our approach yields more coherent and reliable edits than single-step or rule-based multistep baselines.

Problem

• image editing
• multi-step instructions
• abstract editing tasks
• open-ended editing
• instruction following

Innovation

• experiential learning
• planning and orchestration
• vision-language reward
• atomic decomposition
• long-horizon image editing