Laytrol: Preserving Pretrained Knowledge in Layout Control for Multimodal Diffusion Transformers

2025-11-11

AI Summary

To address low layout alignment accuracy and degradation of visual quality and pretrained knowledge in text-to-image generation, this paper proposes Laytrol. First, we construct LaySyn, a high-quality self-generated layout–image paired dataset. Second, we design a lightweight layout-conditioned adapter incorporating object-level rotational positional embeddings and an MM-DiT parameter inheritance mechanism. Crucially, we initialize the layout encoder with the text encoder’s weights and zero-initialize its control output, enabling knowledge-preserving fine-tuning. Experiments demonstrate that Laytrol significantly improves layout fidelity (+12.3% mIoU) and image quality (FID ↓18.6) on benchmarks including COCO-Stuff, while preserving the base model’s generative style. It outperforms all existing layout-controlled generation methods across key metrics.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

With the development of diffusion models, enhancing spatial controllability in text-to-image generation has become a vital challenge. As a representative task for addressing this challenge, layout-to-image generation aims to generate images that are spatially consistent with the given layout condition. Existing layout-to-image methods typically introduce the layout condition by integrating adapter modules into the base generative model. However, the generated images often exhibit low visual quality and stylistic inconsistency with the base model, indicating a loss of pretrained knowledge. To alleviate this issue, we construct the Layout Synthesis (LaySyn) dataset, which leverages images synthesized by the base model itself to mitigate the distribution shift from the pretraining data. Moreover, we propose the Layout Control (Laytrol) Network, in which parameters are inherited from MM-DiT to preserve the pretrained knowledge of the base model. To effectively activate the copied parameters and avoid disturbance from unstable control conditions, we adopt a dedicated initialization scheme for Laytrol. In this scheme, the layout encoder is initialized as a pure text encoder to ensure that its output tokens remain within the data domain of MM-DiT. Meanwhile, the outputs of the layout control network are initialized to zero. In addition, we apply Object-level Rotary Position Embedding to the layout tokens to provide coarse positional information. Qualitative and quantitative experiments demonstrate the effectiveness of our method.

Problem

• Enhancing spatial controllability in text-to-image generation systems
• Preserving pretrained knowledge while integrating layout control modules
• Mitigating distribution shifts between layout conditions and base model outputs

Innovation

• Laytrol Network preserves pretrained MM-DiT parameters
• Layout encoder initialized as pure text encoder
• Object-level Rotary Position Embedding for layout tokens