Mural: Transferring LLM knowledge to image generation via Mixture-of-Transformers

2026-06-27

AI Summary

This work addresses the challenge of effectively leveraging the knowledge of a frozen large language model (LLM) for text-to-image generation using only standard text–image pairs for training. To this end, the authors propose a Mixture-of-Transformers architecture that couples a frozen, reasoning-capable LLM with a diffusion-based image generator through shared attention mechanisms, enabling knowledge transfer without multimodal joint training or explicit reasoning supervision. This approach achieves, for the first time, frozen LLM-guided image synthesis and demonstrates emergent capabilities absent from the training data—including cross-lingual generation, color-guided composition, and scene construction from emojis or ASCII art. The model attains strong performance across multiple benchmarks, including GenEval (0.85), DPG-Bench (86.75), and WISE (0.66).

Abstract

Leveraging capabilities of large language models (LLMs) in text-to-image (T2I) synthesis is an important research direction. In this work we investigate whether the knowledge of a frozen LLM can be effectively utilized in T2I generation when trained exclusively on standard text-image pairs. We integrate a frozen, reasoning-capable LLM with a diffusion-based image generator via shared attention within the Mixture-of-Transformers (MoT) architecture. Our experiments span two critical questions: (1) what degree of the LLM's intrinsic knowledge remains accessible during T2I training, and (2) what novel capabilities emerge in the resulting system. Across established benchmarks, our models achieve strong performance among unified understanding-generation systems: 0.85 on GenEval, 86.75 on DPG-Bench, and 0.66 on WISE with inference-time reasoning, using only text-image data. Remarkably, we uncover emergent behaviors absent from training data, including cross-lingual image generation, color-guided composition, emoji / ASCII scene construction, and generation directed by world knowledge. These results demonstrate that pretrained LLM knowledge can guide image synthesis under standard text-to-image training paradigms, without interleaved multimodal signals or explicit reasoning supervision. Our findings open new avenues for harnessing frozen model capabilities in resource-constrained multimodal learning.

Problem

• text-to-image generation
• large language models
• knowledge transfer
• frozen LLM
• multimodal learning

Innovation

• Mixture-of-Transformers
• frozen LLM
• text-to-image generation
• emergent capabilities
• knowledge transfer