Problem
Research questions and friction points this paper is trying to address.
sketch-based image editing
pixel-level precision
local deformation
geometric constraints
semantic instructions
SI-Edit: Toward Sketch-Instruction Guided Local Image Editing with Pixel-Level Precision
🤖 AI Summary
This work addresses the challenge of achieving pixel-level precision in sketch-based image editing, which is hindered by the absence of high-quality datasets that jointly encode geometric constraints and semantic instructions. To overcome this limitation, we propose SI-Edit, a novel framework accompanied by SI-Data—the first high-quality dataset specifically designed for instruction-guided sketch editing. SI-Data comprises quadruplets of images, sketches, natural language instructions, and corresponding edited results, automatically generated using multimodal large language models to enable spatial-semantic co-learning. By jointly modeling geometric sketches and semantic instructions, our method achieves high-fidelity local deformations and significantly outperforms existing approaches in both structural preservation and alignment with user intent.
📝 Abstract
Despite rapid advances in generative models, achieving pixel-level precision in sketch-based image editing remains a persistent challenge, particularly for fine-grained local deformations. This gap stems primarily from the critical shortage of high-quality, publicly available benchmark datasets that jointly provide geometric constraints and semantic instructions. To address this issue, we first introduce **SI-Data**, a high-quality dataset specifically designed for instruction-guided local sketch editing. We develop an automated pipeline leveraging Multimodal Large Language Models (MLLMs) to synthesize comprehensive quadruplets comprising original images, local geometric sketches, semantic instructions, and corresponding edited images. By providing both reliable spatial anchors and explicit semantic intent, SI-Data uniquely enables collaborative spatial-semantic learning. Building upon this, we propose a collaborative framework called **SI-Edit** that integrates semantic instructions with precise geometric constraints. Furthermore, to address the lack of standardized evaluation, we establish a comprehensive set of metrics designed to measure both structural fidelity (e.g., sketch-to-edge alignment) and semantic adherence. Experimental results demonstrate that SI-Edit provides more reliable structural control than baselines for sketch-based image editing, and achieves precise, pixel-level local refinements aligned with user intent. The data and code are released on the [project page](https://github.com/ywxsuperstar/SIEdit).
Innovation
Methods, ideas, or system contributions that make the work stand out.
sketch-instruction guided editing
pixel-level precision
multimodal large language models
local image editing
spatial-semantic learning
🔎 Similar Papers
No similar papers found.
