Sketch-to-Layout: Sketch-Guided Multimodal Layout Generation

2025-10-31

AI Summary

Existing layout generation methods rely on complex formal constraints, resulting in poor user-friendliness; sketch-based interaction—despite its intuitive appeal—has not been systematically explored for layout synthesis. Method: This paper formally introduces the novel task of “sketch-to-layout generation” and proposes a multimodal Transformer-based framework that jointly models hand-drawn sketches and content assets. To address data scarcity, we design a principled synthetic data generation strategy, producing over 200,000 high-quality sketch-layout pairs. Contribution/Results: Our method achieves significant improvements over state-of-the-art constraint-based layout approaches on PubLayNet, DocLayNet, and SlidesVQA. This work establishes sketch-driven layout generation as a new paradigm, enabling more intuitive and flexible design workflows. To foster community advancement, we publicly release both the synthetic dataset and the trained model.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Game Design — Procedural Content Generation & Storytelling
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Scalable techniques for the creation, curation, publication, maintenance, and consumption of large, Web-based, structured, reusable, knowledge graphs and ontologies

Abstract

Graphic layout generation is a growing research area focusing on generating aesthetically pleasing layouts ranging from poster designs to documents. While recent research has explored ways to incorporate user constraints to guide the layout generation, these constraints often require complex specifications which reduce usability. We introduce an innovative approach exploiting user-provided sketches as intuitive constraints and we demonstrate empirically the effectiveness of this new guidance method, establishing the sketch-to-layout problem as a promising research direction, which is currently under-explored. To tackle the sketch-to-layout problem, we propose a multimodal transformer-based solution using the sketch and the content assets as inputs to produce high quality layouts. Since collecting sketch training data from human annotators to train our model is very costly, we introduce a novel and efficient method to synthetically generate training sketches at scale. We train and evaluate our model on three publicly available datasets: PubLayNet, DocLayNet and SlidesVQA, demonstrating that it outperforms state-of-the-art constraint-based methods, while offering a more intuitive design experience. In order to facilitate future sketch-to-layout research, we release O(200k) synthetically-generated sketches for the public datasets above. The datasets are available at https://github.com/google-deepmind/sketch_to_layout.

Problem

• Generating layouts using intuitive sketch constraints
• Reducing complex specification requirements for layout design
• Creating synthetic sketch training data for model scalability

Innovation

• Uses sketches as intuitive constraints for layout generation
• Employs multimodal transformer with sketch and content inputs
• Generates synthetic sketch training data at scale