MMCORE: MultiModal COnnection with Representation Aligned Latent Embeddings

2026-04-21

AI Summary

This work addresses the challenge of efficiently integrating semantic understanding and visual synthesis in multimodal image generation and editing. The authors propose a novel approach that avoids autoregressive mechanisms or extensive coupling with—and retraining of—diffusion models. Instead, learnable query tokens are introduced to extract semantic visual embeddings from a pretrained vision-language model, which are then injected as conditioning signals into a diffusion model. This enables high-quality text-to-image generation, interleaved image synthesis, and editing. The method consistently outperforms current state-of-the-art approaches across multiple generation and editing benchmarks, significantly enhancing the synergy between multimodal semantic comprehension and visual synthesis.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We present MMCORE, a unified framework designed for multimodal image generation and editing. MMCORE leverages a pre-trained Vision-Language Model (VLM) to predict semantic visual embeddings via learnable query tokens, which subsequently serve as conditioning signals for a diffusion model. This streamlined design effectively transfers the rich understanding and reasoning capabilities of VLMs into the visual generation process. By obviating the need for deep fusion between autoregressive and diffusion models or training from scratch, MMCORE significantly reduces computational overhead while maintaining high-fidelity synthesis. MMCORE seamlessly integrates text-to-image synthesis with interleaved image generation, demonstrating robust multimodal comprehension in complex scenarios such as spatial reasoning and visual grounding. Comprehensive evaluations indicate that MMCORE consistently outperforms state-of-the-art baselines across a broad spectrum of text-to-image and single/multi-image editing benchmarks.

Problem

• multimodal image generation
• vision-language model
• diffusion model
• image editing
• semantic visual embeddings

Innovation

• multimodal generation
• vision-language model
• latent embedding alignment
• diffusion conditioning
• image editing