Rethinking Scribble-Guided Image Editing: Generalization, Instruction Adherence, and Multi-Tasking

2026-05-25

AI Summary

This work addresses the limitations of current scribble-guided image editing models, which suffer from poor instruction generalization and inconsistent editing results in multi-task scenarios. Through empirical analysis, the study identifies instruction-level generalization as a key bottleneck and proposes a three-stage optimization strategy: first, a curriculum learning mechanism prioritizing coverage before fidelity; second, a zero-cost multi-task mosaic construction method to enhance task diversity; and third, a region-focused loss function to improve local consistency. By integrating synthetic and real data through a two-stage training pipeline and employing multi-task sample stitching, the proposed model significantly outperforms existing approaches on the VIBE benchmark. The authors also release their dataset and code to support reproducibility and further research.

Abstract

Scribble-guided image editing allows users to combine simple scribble annotations with text prompts to specify both where and how an image should be edited, enabling flexible interaction with precise spatial control. However, existing models still exhibit unstable performance under this paradigm, especially in multi-task scenarios. To improve performance, we conduct empirical studies using an open-source editing model and reveal an asymmetry in generalization: instruction-level generalization, including across editing tasks and from single-task to multi-task settings, is more challenging than image-domain generalization, such as from synthetic to real-world images or from mosaicked to regular images. This suggests that the primary bottleneck lies in insufficient learning for diverse editing instructions rather than in the image domain gap. Motivated by this insight, we propose three strategies: (a) a Coverage-then-Realism Curriculum, a two-stage pipeline that first builds large-scale synthetic, instruction-rich data for broad task supervision, then curates a small set of real-world data to refine generation realism; (b) Multi-Task Mosaicking, which constructs multi-task training samples by concatenating single-task examples at nearly zero cost while enabling the learned capability to generalize to non-mosaicked images; and (c) an Edit-Focused Loss, which leverages the changed regions between input and output images in synthetic data to focus training on edited regions, improving both learning efficiency and editing accuracy. With these strategies, we substantially improve both single-task and multi-task scribble-guided editing on the VIBE benchmark, achieving state-of-the-art results. We will publicly release our dataset and model.

Problem

• scribble-guided image editing
• instruction adherence
• multi-tasking
• generalization
• image editing

Innovation

• scribble-guided editing
• instruction generalization
• multi-task learning
• curriculum learning
• edit-focused loss