Bifrost-1: Bridging Multimodal LLMs and Diffusion Models with Patch-level CLIP Latents

2025-08-07

AI Summary

This work addresses the degradation of reasoning capabilities in multimodal large language models (MLLMs) when integrated with high-fidelity visual generation. We propose a lightweight fusion framework that avoids retraining the LLM backbone: leveraging patch-level CLIP embeddings as a unified visual representation bridge, we design a dual-branch MLLM architecture incorporating a ControlNet fine-tuning module and a diffusion-based generation branch. This decouples multimodal comprehension from controllable image synthesis, preserving the model’s original complex reasoning abilities while significantly improving generation fidelity and controllability. Experiments demonstrate that our method achieves superior or competitive image fidelity and multimodal understanding performance compared to state-of-the-art approaches, at substantially lower training cost. Ablation studies validate the effectiveness of each component.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Diffusion Models for Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

There is growing interest in integrating high-fidelity visual synthesis capabilities into large language models (LLMs) without compromising their strong reasoning capabilities. Existing methods that directly train LLMs or bridge LLMs and diffusion models usually suffer from costly training since the backbone LLMs have not seen image representations during pretraining. We present Bifrost-1, a unified framework that bridges pretrained multimodal LLMs (MLLMs) and diffusion models using patch-level CLIP image embeddings as latent variables, which are natively aligned with the MLLM's CLIP visual encoder. These patch-level image embeddings are integrated into the diffusion model with a lightweight adaptation of its ControlNet. To retain the original multimodal reasoning capabilities of MLLMs, we equip the MLLM with a visual generation branch initialized from the original MLLM parameters when predicting the patch-level image embeddings. By seamlessly integrating pretrained MLLMs and diffusion models with patch-level CLIP latents, our framework enables high-fidelity controllable image generation with significant training efficiency. Our experiments demonstrate that Bifrost-1 achieves comparable or better performance than previous methods in terms of visual fidelity and multimodal understanding, with substantially lower compute during training. We also provide comprehensive ablation studies showing the effectiveness of our design choices.

Problem

• Integrate visual synthesis into LLMs without losing reasoning
• Reduce costly training of LLMs with image representations
• Enable high-fidelity controllable image generation efficiently

Innovation

• Uses patch-level CLIP latents for alignment
• Lightweight ControlNet adapts diffusion model
• Visual generation branch retains MLLM capabilities