Multimodal Markup Document Models for Graphic Design Completion

2024-09-27

arXiv.org

AI Summary

Existing element-attribute grid representations for graphic design completion tasks struggle to model variable-length, type-heterogeneous, and multimodal (text-image) structures. Method: We propose a unified interleaved multimodal tokenized document model that jointly encodes syntactic and semantic structures of markup languages (e.g., SVG/HTML) alongside variable-size, alpha-channel-aware local image generation. We introduce a specialized image quantizer for efficient transparent-image tokenization and integrate an enhanced code-large language model with an interleaved multimodal sequence architecture. Contribution/Results: Our model achieves significant improvements over baselines on three design completion tasks—missing template attributes, image synthesis, and text generation—demonstrating its effectiveness in jointly modeling structural logic and visual semantics in design documents.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This paper presents multimodal markup document models (MarkupDM) that can generate both markup language and images within interleaved multimodal documents. Unlike existing vision-and-language multimodal models, our MarkupDM tackles unique challenges critical to graphic design tasks: generating partial images that contribute to the overall appearance, often involving transparency and varying sizes, and understanding the syntax and semantics of markup languages, which play a fundamental role as a representational format of graphic designs. To address these challenges, we design an image quantizer to tokenize images of diverse sizes with transparency and modify a code language model to process markup languages and incorporate image modalities. We provide in-depth evaluations of our approach on three graphic design completion tasks: generating missing attribute values, images, and texts in graphic design templates. Results corroborate the effectiveness of our MarkupDM for graphic design tasks. We also discuss the strengths and weaknesses in detail, providing insights for future research on multimodal document generation.

Problem

• Completes missing parts in graphic design documents
• Unifies various design tasks through multimodal representation
• Generates images and text for design automation

Innovation

• Multimodal markup document model for design
• Fill-in-the-middle training for unified tasks
• Discrete image token prediction with transparency support