RepFusion: Leveraging Multimodal Priors for Denoising in Representation Space

2026-06-12

AI Summary

This work proposes a novel framework that leverages multimodal large language models (MLLMs) beyond their conventional role as text encoders in text-to-image generation. Specifically, the MLLM is employed as an encoder for noisy visual representations, with its outputs serving as conditioning signals for a diffusion Transformer to perform efficient denoising in the representation space. By iteratively invoking the MLLM and integrating a representation autoencoder (RAE), the method effectively harnesses the strong semantic priors embedded in the MLLM to enhance image generation quality. Experimental results demonstrate that, under comparable inference costs, the proposed approach significantly outperforms baseline models employing similarly sized, newly initialized denoisers, thereby validating the critical role of MLLMs in visual denoising.

Abstract

Large language models (LLMs) are widely used in text-to-image (T2I) systems, but they are typically limited to text encoding, while denoising is handled by newly trained generative backbones. The emergence of representation autoencoders (RAEs) shifts the generation target toward semantically structured visual representations, creating a latent space that is more compatible with pretrained LLM priors. Inspired by multimodal LLMs (MLLMs), where an MLP projector is sufficient to align clean visual representations with a pretrained LLM, we repurpose the MLLM itself as a noisy representation encoder, extending this mechanism from clean to noisy inputs. We present RepFusion, which uses the resulting MLLM outputs as the conditioning signal for a diffusion transformer. In controlled comparisons at similar inference budgets, RepFusion outperforms baselines that devote comparable capacity to newly initialized denoisers. These results demonstrate that MLLMs provide strong priors for denoising visual representations and that, by conditioning on evolving noisy representations, test-time compute can be productively spent on repeated MLLM conditioning in modern T2I systems.

Problem

• denoising
• representation space
• multimodal LLMs
• text-to-image generation
• visual representations

Innovation

• RepFusion
• multimodal LLMs
• representation autoencoders
• denoising in representation space
• diffusion transformer