GlyphDraw2: Automatic Generation of Complex Glyph Posters with Diffusion Models and Large Language Models

2024-07-02

arXiv.org

AI Summary

Existing methods struggle to generate high-resolution, bilingual (Chinese–English) posters with natural text–image integration and typographic fidelity. To address this, we propose the first end-to-end poster generation framework enabling joint semantic, typographic, and layout control. Our approach features: (1) a novel triple cross-attention mechanism that enables fine-grained alignment among textual content, font attributes, and background context; (2) the first large-scale, high-resolution (≥1024p), bilingual font–poster paired dataset; and (3) a hybrid architecture integrating SDXL diffusion models with large language models (LLMs) to support controllable font selection, precise text rendering, and multi-resolution (≥1024×1024) high-fidelity output. Extensive experiments demonstrate significant improvements in text accuracy and visual consistency, achieving state-of-the-art performance on complex-background poster generation tasks.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Language Grounding & Multi-modal NLP
• Computer Vision: Diffusion Models for Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Posters play a crucial role in marketing and advertising by enhancing visual communication and brand visibility, making significant contributions to industrial design. With the latest advancements in controllable T2I diffusion models, increasing research has focused on rendering text within synthesized images. Despite improvements in text rendering accuracy, the field of automatic poster generation remains underexplored. In this paper, we propose an automatic poster generation framework with text rendering capabilities leveraging LLMs, utilizing a triple-cross attention mechanism based on alignment learning. This framework aims to create precise poster text within a detailed contextual background. Additionally, the framework supports controllable fonts, adjustable image resolution, and the rendering of posters with descriptions and text in both English and Chinese.Furthermore, we introduce a high-resolution font dataset and a poster dataset with resolutions exceeding 1024 pixels. Our approach leverages the SDXL architecture. Extensive experiments validate our method's capability in generating poster images with complex and contextually rich backgrounds.Codes is available at https://github.com/OPPO-Mente-Lab/GlyphDraw2.

Problem

• Automated Poster Design
• Text-Image Fusion
• Multilingual Typography

Innovation

• Supercomputer Modeling
• Language Understanding AI
• SDXL Poster Generation