Neural Mutual Information Estimation with Vector Copulas

2025-10-23

AI Summary

Mutual information (MI) estimation is fundamental in data science, yet existing approaches struggle to reconcile model flexibility with statistical interpretability: neural estimators require substantial data, while classical models (e.g., Gaussian copulas) fail to capture complex, high-order dependencies. This paper introduces the first neural MI estimation framework grounded in vector vine structures—marking the first integration of vector vine theory into deep learning–based estimation. Methodologically, we employ neural networks to parameterize marginal transformations and leverage vector vines to explicitly model intricate multivariate dependence structures; training is performed end-to-end via a variational lower bound and density-ratio estimation. Evaluated on synthetic benchmarks and multimodal real-world datasets, our approach achieves significant gains in estimation accuracy, robustness, and generalization—effectively balancing expressive power and statistical interpretability.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Multi-modal Vision

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Estimating mutual information (MI) is a fundamental task in data science and machine learning. Existing estimators mainly rely on either highly flexible models (e.g., neural networks), which require large amounts of data, or overly simplified models (e.g., Gaussian copula), which fail to capture complex distributions. Drawing upon recent vector copula theory, we propose a principled interpolation between these two extremes to achieve a better trade-off between complexity and capacity. Experiments on state-of-the-art synthetic benchmarks and real-world data with diverse modalities demonstrate the advantages of the proposed estimator.

Problem

• Estimating mutual information in data science and machine learning
• Addressing limitations of existing overly complex or simplified estimators
• Achieving better trade-off between model complexity and capacity

Innovation

• Vector copula theory enables principled mutual information interpolation
• Balances complexity and capacity between neural and Gaussian models
• Outperforms existing estimators on synthetic and real-world data