Making Image Editing Easier via Adaptive Task Reformulation with Agentic Executions

2026-04-17

AI Summary

Existing image editing methods often exhibit instability when handling small objects, ambiguous instructions, or complex spatial relationships, primarily due to a mismatch between task formulation and model capabilities. This work proposes an adaptive task restructuring framework that leverages multimodal large language model (MLLM) agents to dynamically analyze input image–instruction pairs. Through task routing, instruction reformulation, and feedback-driven iterative refinement, the framework translates the original request into a more executable sequence of operations. Notably, it achieves substantial performance gains without modifying the underlying editing model. The approach consistently outperforms baseline methods across multiple benchmarks—including ImgEdit, PICA, and RePlan—with particularly pronounced improvements on challenging cases, thereby demonstrating the critical role of task restructuring in aligning user intent with the effective operating regime of editing models.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Instruction guided image editing has advanced substantially with recent generative models, yet it still fails to produce reliable results across many seemingly simple cases. We observe that a large portion of these failures stem not from insufficient model capacity, but from poorly formulated editing tasks, such as those involving small targets, implicit spatial relations, or under-specified instructions. In this work, we frame image editing failures as a task formulation problem and propose an adaptive task reformulation framework that improves editing performance without modifying the underlying model. Our key idea is to transform the original image-instruction pair into a sequence of operations that are dynamically determined and executed by a MLLM agent through analysis, routing, reformulation, and feedback-driven refinement. Experiments on multiple benchmarks, including ImgEdit, PICA, and RePlan, across diverse editing backbones such as Qwen Image Edit and Nano Banana, show consistent improvements, with especially large gains on challenging cases. These results suggest that task reformulation is a critical but underexplored factor, and that substantial gains can be achieved by better matching editing tasks to the effective operating regime of existing models.

Problem

• image editing
• task formulation
• instruction ambiguity
• spatial relations
• editing reliability

Innovation

• adaptive task reformulation
• agentic execution
• instruction-guided image editing
• MLLM agent
• task formulation