TextFlux: An OCR-Free DiT Model for High-Fidelity Multilingual Scene Text Synthesis

2025-05-23

AI Summary

To address key limitations in multilingual scene text synthesis—including reliance on OCR modules, heavy dependence on large-scale annotated data, and inflexible layout control—this paper proposes the first OCR-free end-to-end Diffusion Transformer (DiT) framework. Instead of employing explicit visual conditional encoders, our method jointly models textual semantics and scene context through the denoising process, enabling accurate glyph generation, high-fidelity rendering, and natural integration into complex backgrounds. Methodologically, it supports low-resource language adaptation with fewer than 1,000 samples and achieves comparable performance using only 1% of standard training data. Moreover, it enables fine-grained, line-level controllable multi-line text synthesis. Extensive qualitative and quantitative evaluations demonstrate that our approach consistently outperforms state-of-the-art methods, delivering substantial improvements in multilingual synthesis quality, layout flexibility, and cross-lingual generalization.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Sentence-level Semantics, Textual Inference, etc.
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Diffusion-based scene text synthesis has progressed rapidly, yet existing methods commonly rely on additional visual conditioning modules and require large-scale annotated data to support multilingual generation. In this work, we revisit the necessity of complex auxiliary modules and further explore an approach that simultaneously ensures glyph accuracy and achieves high-fidelity scene integration, by leveraging diffusion models' inherent capabilities for contextual reasoning. To this end, we introduce TextFlux, a DiT-based framework that enables multilingual scene text synthesis. The advantages of TextFlux can be summarized as follows: (1) OCR-free model architecture. TextFlux eliminates the need for OCR encoders (additional visual conditioning modules) that are specifically used to extract visual text-related features. (2) Strong multilingual scalability. TextFlux is effective in low-resource multilingual settings, and achieves strong performance in newly added languages with fewer than 1,000 samples. (3) Streamlined training setup. TextFlux is trained with only 1% of the training data required by competing methods. (4) Controllable multi-line text generation. TextFlux offers flexible multi-line synthesis with precise line-level control, outperforming methods restricted to single-line or rigid layouts. Extensive experiments and visualizations demonstrate that TextFlux outperforms previous methods in both qualitative and quantitative evaluations.

Problem

• OCR-free multilingual scene text synthesis without visual conditioning modules
• High-fidelity text generation with minimal training data in low-resource languages
• Controllable multi-line text synthesis surpassing single-line layout limitations

Innovation

• OCR-free DiT model for multilingual text
• Strong multilingual scalability with minimal data
• Controllable multi-line text generation flexibility