Self-Prompting Diffusion Transformer for Open-Vocabulary Scene Text Editing via In-Context Learning

2026-05-15

AI Summary

This work addresses the limitations of existing scene text editing methods, which struggle to support open-vocabulary editing and often fail to preserve the original visual style due to their reliance on pretrained glyph encoders. To overcome these challenges, the authors propose a self-prompting-based text editing approach that leverages the in-context learning capability of multimodal diffusion Transformers (MM-DiT) to directly extract style and glyph cues from the input image—eliminating the need for external encoders. This enables stylistically consistent editing for arbitrary vocabulary and multiple languages. Combined with a self-supervised pretraining scheme and a two-stage fine-tuning strategy, the method achieves state-of-the-art performance across multiple benchmarks, significantly improving both text accuracy and style fidelity.

Abstract

Scene text editing aims to modify text in a target region of an image while preserving surrounding background style and texture. Existing methods rely solely on image background information while neglecting the visual details of target regions, which discards stylistic features in the original text and essentially degrades the task to text rendering. Moreover, the conditions imposed by pre-trained glyph encoder limit the scope of editable text. To address these issues, this paper proposes a self-prompting scene text editing method that constructs style and glyph prompts directly from the original image, without introducing additional style or glyph encoders. We employ a two-stage training strategy: the diffusion transformer is first trained on large-scale self-supervised data and then refined using a small set of paired images. By leveraging the in-context learning capability of the Multi-Modal Diffusion Transformer (MM-DiT), it achieves open-vocabulary and style-consistent text editing. Experimental results on various languages demonstrate that our method achieves the state-of-the-art performance in both text accuracy and style consistency. Our project page: \href{https://hongxiii.github.io/mstedit}{hongxiii.github.io/mstedit}.

Problem

• scene text editing
• style consistency
• open-vocabulary
• glyph representation
• visual detail preservation

Innovation

• Self-Prompting
• Diffusion Transformer
• In-Context Learning
• Open-Vocabulary Text Editing
• Style Consistency