Text-Vision Co-Instructed Image Editing

2026-06-15

AI Summary

This work addresses the challenge that existing image editing methods struggle to simultaneously achieve strong semantic expressiveness from textual instructions and precise spatial control from visual cues. To bridge this gap, the authors propose the TV-Edit framework, which introduces a novel dataset comprising over 23K paired text–visual instruction samples. This dataset enables the semantic interpretation of sparse visual guidance—such as drag-and-drop or point-based inputs—and integrates it into a joint image–text context to generate semantically aware control signals. The study also establishes TV-Edit-Bench, a new evaluation benchmark for text–visual editing. Experiments demonstrate that TV-Edit significantly outperforms text-only or vision-only baselines across multiple editing backbones, achieving state-of-the-art performance in semantic fidelity, spatial alignment, and structural consistency.

Abstract

Existing image editing methods can be generally categorized into textual instruction-based and visual prompt-based ones. Textual instructions are semantically expressive, but are limited by the coarse granularity of spatial control of the editing results. In contrast, visual prompts such as drag and point can provide precise spatial guidance, but are limited by the inherent ambiguity in semantic intent. To unify the strength of textual and visual prompts, we present Text-Vision Co-Instructed Image Editing, which jointly models textual instructions as semantic intent and sparse visual instructions as spatial guidance, aiming to achieve precise and intent-faithful image manipulation. To this end, we first construct a textual-visual instruction paired dataset with more than 23K samples derived from dynamic videos, enabling aligned supervision for cross-modal instruction. We then propose TV-Edit, a Textual-Visual instruction unified Editing framework to contextualize drag or point-based visual instructions with image-text semantics and lift them into semantic-aware control representations for pretrained editing backbones. By integrating semantic intent and spatial constraints, TV-Edit leads to more precise spatial control, less instruction ambiguity, and stronger structural consistency than text-only or drag-based alternatives. Finally, we establish TV-Edit-Bench, a deliberately designed benchmark to evaluate semantic faithfulness, spatial alignment, and visual consistency with ground-truth references and controlled textual-visual variations for reliable assessment. Our experiments across multiple editing backbones demonstrate that TV-Edit consistently yields more precise and intent-faithful edits, significantly outperforming state-of-the-art instruction-based and drag-based baselines.

Problem

• image editing
• textual instruction
• visual prompt
• spatial control
• semantic intent

Innovation

• text-vision co-instruction
• semantic-aware control
• spatial guidance
• image editing
• cross-modal alignment