CoVAE: correlated multimodal generative modeling

2026-03-02

AI Summary

Existing multimodal variational autoencoders often disrupt the joint statistical structure across modalities during latent space fusion, leading to degraded generation quality and inaccurate uncertainty quantification. This work proposes CoVAE, which, for the first time within the variational autoencoder framework, explicitly models inter-modal correlations by introducing a correlation-constrained generative mechanism. This approach preserves the underlying joint distributional structure while enabling high-quality cross-modal reconstruction and reliable uncertainty estimation. Experimental results demonstrate that CoVAE significantly outperforms current state-of-the-art methods across multiple real-world and synthetic datasets.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Multimodal Variational Autoencoders have emerged as a popular tool to extract effective representations from rich multimodal data. However, such models rely on fusion strategies in latent space that destroy the joint statistical structure of the multimodal data, with profound implications for generation and uncertainty quantification. In this work, we introduce Correlated Variational Autoencoders (CoVAE), a new generative architecture that captures the correlations between modalities. We test CoVAE on a number of real and synthetic data sets demonstrating both accurate cross-modal reconstruction and effective quantification of the associated uncertainties.

Problem

• multimodal generative modeling
• joint statistical structure
• cross-modal reconstruction
• uncertainty quantification
• latent space fusion

Innovation

• Correlated Variational Autoencoders
• multimodal generative modeling
• cross-modal reconstruction
• uncertainty quantification
• joint statistical structure