Learning Multimodal Energy-Based Model with Multimodal Variational Auto-Encoder via MCMC Revision

2026-05-01

AI Summary

This work addresses the challenge of slow mixing in Markov Chain Monte Carlo (MCMC) sampling for multimodal energy-based models (EBMs) trained via maximum likelihood, particularly due to poor cross-modal consistency when using conventional noise-initialized Langevin dynamics. To overcome this limitation, the authors propose a joint learning framework that alternates between maximum likelihood estimation and MCMC refinement in both data and latent spaces. The framework co-trains a multimodal EBM, a shared-latent generative model, and a joint inference network. The generator supplies high-quality initial samples for MCMC, while the inference model introduces a flexible latent-space prior that relaxes restrictive Gaussian assumptions. Experimental results demonstrate that the proposed approach significantly outperforms existing baselines in terms of multimodal synthesis quality and cross-modal consistency, confirming its effectiveness and scalability.

Abstract

Energy-based models (EBMs) are a flexible class of deep generative models and are well-suited to capture complex dependencies in multimodal data. However, learning multimodal EBM by maximum likelihood requires Markov Chain Monte Carlo (MCMC) sampling in the joint data space, where noise-initialized Langevin dynamics often mixes poorly and fails to discover coherent inter-modal relationships. Multimodal VAEs have made progress in capturing such inter-modal dependencies by introducing a shared latent generator and a joint inference model. However, both the shared latent generator and joint inference model are parameterized as unimodal Gaussian (or Laplace), which severely limits their ability to approximate the complex structure induced by multimodal data. In this work, we study the learning problem of the multimodal EBM, shared latent generator, and joint inference model. We present a learning framework that effectively interweaves their MLE updates with corresponding MCMC refinements in both the data and latent spaces. Specifically, the generator is learned to produce coherent multimodal samples that serve as strong initial states for EBM sampling, while the inference model is learned to provide informative latent initializations for generator posterior sampling. Together, these two models serve as complementary models that enable effective EBM sampling and learning, yielding realistic and coherent multimodal EBM samples. Extensive experiments demonstrate superior performance for multimodal synthesis quality and coherence compared to various baselines. We conduct various analyses and ablation studies to validate the effectiveness and scalability of the proposed multimodal framework.

Problem

• multimodal energy-based model
• Markov Chain Monte Carlo
• multimodal variational auto-encoder
• cross-modal coherence
• joint data space

Innovation

• Multimodal Energy-Based Model
• MCMC Revision
• Shared Latent Generator
• Joint Inference Model
• Coherent Multimodal Synthesis