WeEdit: A Dataset, Benchmark and Glyph-Guided Framework for Text-centric Image Editing

2026-03-12

AI Summary

This work addresses the limitations of existing image-based text editing methods, which often suffer from blurry outputs or hallucinations due to the absence of dedicated training paradigms, large-scale datasets, and standardized evaluation protocols. To overcome these challenges, the authors propose the first closed-loop system specifically designed for text editing in images. The approach leverages HTML to automatically construct a scalable multilingual training dataset and introduces a two-stage training strategy: first, glyph-guided fine-tuning injects spatial and content priors; second, multi-objective reinforcement learning jointly optimizes instruction following, text clarity, and background preservation. Furthermore, the study establishes the first standardized bilingual and multilingual benchmark for text editing evaluation. Experimental results demonstrate that the proposed method significantly outperforms existing open-source models across diverse editing tasks, producing more accurate and legible text while better preserving background integrity, thereby validating its effectiveness and generalization capability.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Instruction-based image editing aims to modify specific content within existing images according to user-provided instructions while preserving non-target regions. Beyond traditional object- and style-centric manipulation, text-centric image editing focuses on modifying, translating, or rearranging textual elements embedded within images. However, existing leading models often struggle to execute complex text editing precisely, frequently producing blurry or hallucinated characters. We attribute these failures primarily to the lack of specialized training paradigms tailored for text-centric editing, as well as the absence of large-scale datasets and standardized benchmarks necessary for a closed-loop training and evaluation system. To address these limitations, we present WeEdit, a systematic solution encompassing a scalable data construction pipeline, two benchmarks, and a tailored two-stage training strategy. Specifically, we propose a novel HTML-based automatic editing pipeline, which generates 330K training pairs covering diverse editing operations and 15 languages, accompanied by standardized bilingual and multilingual benchmarks for comprehensive evaluation. On the algorithmic side, we employ glyph-guided supervised fine-tuning to inject explicit spatial and content priors, followed by a multi-objective reinforcement learning stage to align generation with instruction adherence, text clarity, and background preservation. Extensive experiments demonstrate that WeEdit outperforms previous open-source models by a clear margin across diverse editing operations.

Problem

• text-centric image editing
• instruction-based image editing
• glyph-guided editing
• image editing benchmark
• text hallucination

Innovation

• text-centric image editing
• glyph-guided training
• HTML-based data pipeline
• multilingual benchmark
• reinforcement learning for image editing