UniLayDiff: A Unified Diffusion Transformer for Content-Aware Layout Generation

2025-12-09

AI Summary

Existing approaches struggle to uniformly model heterogeneous constraints—such as element types, dimensions, and spatial relationships—in content-aware layout generation. Method: We propose the first end-to-end, single-model framework capable of unifying unconditional and fully conditional layout generation across the entire spectrum of constraints. Our approach treats layout constraints as independent modalities and introduces a multimodal diffusion Transformer that jointly encodes background images, element semantics, and relational structures. To enable efficient and scalable joint optimization, we incorporate a LoRA-driven relational fine-tuning mechanism. Results: Experiments demonstrate state-of-the-art performance across diverse constraint settings. Notably, our model is the first to comprehensively cover the full task spectrum of content-aware layout generation—spanning unconditional, partially conditional, and fully conditional regimes—thereby establishing a general, flexible, and interpretable foundation for automated graphic design.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Content-aware layout generation is a critical task in graphic design automation, focused on creating visually appealing arrangements of elements that seamlessly blend with a given background image. The variety of real-world applications makes it highly challenging to develop a single model capable of unifying the diverse range of input-constrained generation sub-tasks, such as those conditioned by element types, sizes, or their relationships. Current methods either address only a subset of these tasks or necessitate separate model parameters for different conditions, failing to offer a truly unified solution. In this paper, we propose UniLayDiff: a Unified Diffusion Transformer, that for the first time, addresses various content-aware layout generation tasks with a single, end-to-end trainable model. Specifically, we treat layout constraints as a distinct modality and employ Multi-Modal Diffusion Transformer framework to capture the complex interplay between the background image, layout elements, and diverse constraints. Moreover, we integrate relation constraints through fine-tuning the model with LoRA after pretraining the model on other tasks. Such a schema not only achieves unified conditional generation but also enhances overall layout quality. Extensive experiments demonstrate that UniLayDiff achieves state-of-the-art performance across from unconditional to various conditional generation tasks and, to the best of our knowledge, is the first model to unify the full range of content-aware layout generation tasks.

Problem

• Unifies diverse content-aware layout generation tasks
• Addresses constraints as distinct modality in single model
• Enhances layout quality through unified conditional generation

Innovation

• Unified Diffusion Transformer for diverse layout tasks
• Multi-Modal Diffusion Transformer captures background and constraints interplay
• LoRA fine-tuning integrates relation constraints after pretraining