Beyond Flat Text: Dual Self-inherited Guidance for Visual Text Generation

2025-01-10

AI Summary

Existing text generation models struggle to preserve typographic fidelity and background consistency when rendering text on non-planar (e.g., slanted or curved) surfaces. To address this, we propose STGen—a training-free framework introducing a novel dual-branch latent-space guidance mechanism: “semantic correction” and “structural injection.” The semantic branch enables cross-layout semantic transfer via flattened text latent representations, while the structural branch integrates glyph-level image latent features to maintain geometric consistency. Crucially, the decoupled design ensures zero-shot generalization to arbitrary non-planar text layouts without retraining. Extensive experiments demonstrate that STGen achieves a 12.6% improvement in OCR accuracy and a 38% reduction in FID across diverse curved and slanted scenarios, significantly enhancing both textual accuracy and visual harmony.

Technology Category

• Natural Language Processing: Generation
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Deep Generative Models & Autoencoders

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications

Abstract

In real-world images, slanted or curved texts, especially those on cans, banners, or badges, appear as frequently, if not more so, than flat texts due to artistic design or layout constraints. While high-quality visual text generation has become available with the advanced generative capabilities of diffusion models, these models often produce distorted text and inharmonious text background when given slanted or curved text layouts due to training data limitation. In this paper, we introduce a new training-free framework, STGen, which accurately generates visual texts in challenging scenarios (eg, slanted or curved text layouts) while harmonizing them with the text background. Our framework decomposes the visual text generation process into two branches: (i) extbf{Semantic Rectification Branch}, which leverages the ability in generating flat but accurate visual texts of the model to guide the generation of challenging scenarios. The generated latent of flat text is abundant in accurate semantic information related both to the text itself and its background. By incorporating this, we rectify the semantic information of the texts and harmonize the integration of the text with its background in complex layouts. (ii) extbf{Structure Injection Branch}, which reinforces the visual text structure during inference. We incorporate the latent information of the glyph image, rich in glyph structure, as a new condition to further strengthen the text structure. To enhance image harmony, we also apply an effective combination method to merge the priors, providing a solid foundation for generation. Extensive experiments across a variety of visual text layouts demonstrate that our framework achieves superior accuracy and outstanding quality.

Problem

• Text Generation Models
• Skewed or Curved Characters
• Complex Layout Scenes

Innovation

• STGen
• Semantic Correction
• Structural Injection