MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generation

2026-06-15

AI Summary

Existing cross-modal generation methods are limited by reliance on text-aligned data, fully paired training setups, or deterministic mappings, hindering flexible and consistent arbitrary-to-arbitrary modality synthesis. This work proposes an end-to-end unified multimodal latent diffusion framework that jointly trains modality-specific encoders and decoders with a streaming prior within a shared stochastic latent space. By introducing a variational inference–based routing objective, the approach balances consistency, predictive adequacy, and content minimality. It is the first to extend latent diffusion models to multimodal arbitrary-to-arbitrary generation, supporting both conditional synthesis and unconditional joint sampling. Experiments demonstrate that the method matches or surpasses state-of-the-art baselines in conditional generation on PolyMNIST-Quadrant-Labels and large-scale image–text–audio benchmarks, while achieving significantly improved consistency in unconditional generation compared to existing approaches.

Abstract

We introduce MUNI, an end-to-end multimodal latent diffusion framework for any-to-any generation that unifies subset-conditioned cross-modal generation and unconditional joint sampling through a shared stochastic latent. Existing multimodal generative models are largely LLM-based, which limits leveraging modality-specific generators and requires text-paired data for training. Recent diffusion- and flow-based any-to-any extensions take a different direction but still rely on text-aligned embeddings, fully-paired training, or matched-dimensionality deterministic mappings. MUNI rests on two complementary contributions, one architectural and one in the training objective. First, we extend latent diffusion to multimodal any-to-any generation end-to-end: instead of the standard two-stage recipe that precomputes a frozen latent space and then fits a prior over it, MUNI jointly trains modality-specific encoders, expressive decoders, and a single shared flow-based prior under one objective. Second, we identify that the standard aggregation rules of multimodal variational inference are insufficient once coupled with a learned prior and expressive decoders. A suitable shared latent must simultaneously satisfy coherence across generated modalities, predictive sufficiency of subset latents, and minimality of the latent content. We propose a routed training objective whose structural choices align the latent with these criteria and admit a minimal-sufficiency characterization in the realizable setting. Experiments on PolyMNIST-Quadrant-Labels and a large-scale image-text-audio benchmark show MUNI matching or exceeding the strongest baselines on conditional generation while opening its largest margins on unconditional coherence. Project page: https://muni-proj.github.io/.

Problem

• multimodal generation
• any-to-any generation
• latent coherence
• unconditional joint sampling
• cross-modal generation

Innovation

• multimodal latent diffusion
• any-to-any generation
• shared stochastic latent
• flow-based prior
• routed training objective