LatentUMM: Dual Latent Alignment for Unified Multimodal Models

2026-05-17

AI Summary

This work addresses the functional inconsistency between understanding and generation tasks in existing unified multimodal models, which leads to semantic drift during cross-modal translation. To resolve this issue, the authors propose a dual latent space alignment mechanism that explicitly aligns the mapping transformations entering and exiting the latent space, thereby enforcing bidirectional consistency between generation and re-encoding at both the modality semantics and model capacity levels. Furthermore, they introduce a latent space dynamic stabilization strategy that integrates strong embedding alignment, bidirectional capacity matching, stochastic latent space unfolding, and preference optimization. Evaluated across multiple architectures, the proposed approach significantly enhances multimodal consistency and effectively mitigates semantic drift.

Abstract

Unified multimodal models (UMMs) achieve strong performance in both understanding and generation by learning a shared latent space, yet they often exhibit functional inconsistency between these two capabilities. We observe that this issue does not stem from a lack of shared representations, but from the absence of explicit alignment between the transformations that map into and out of the latent space. As a result, generation and re-encoding can follow inconsistent trajectories, leading to semantic drift under modality transitions. In this work, we propose LatentUMM, a framework that constructs an enhanced shared latent space to explicitly align these transformations and improve cross-modal consistency. LatentUMM consists of two stages. First, dual latent alignment enforces consistency at both the modality and capacity levels: cross-modal alignment uses a stronger embedding model to impose structured cross-modal semantics, while dual capacity alignment enforces bidirectional consistency under generation and re-encoding. Second, latent dynamics stabilization improves robustness via stochastic latent rollouts and preference optimization, favoring trajectories that better preserve semantic consistency. Experiments show that LatentUMM consistently improves multimodal consistency across diverse architectures. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/LatentUMM.

Problem

• unified multimodal models
• functional inconsistency
• latent space alignment
• semantic drift
• cross-modal consistency

Innovation

• dual latent alignment
• unified multimodal models
• cross-modal consistency
• latent dynamics stabilization
• semantic drift