A note on the relations between mixture models, maximum-likelihood and entropic optimal transport

2025-01-21

AI Summary

The maximum likelihood estimation (MLE) for Gaussian mixture models (GMMs) and entropy-regularized optimal transport (Entropic OT) have been studied independently, with no established formal connection between their optimization landscapes. Method: This paper establishes a rigorous functional equivalence: the GMM MLE objective is mathematically identical—in parameter space—to a specific Entropic OT loss. Under this unified framework, the Expectation-Maximization (EM) algorithm is interpreted as block-coordinate descent (BCD) applied to this OT objective. Contribution/Results: We provide the first concise, rigorous, pedagogically accessible derivation of this equivalence, verified both theoretically and empirically via GMM case studies. Key innovations include: (1) formal characterization of the exact functional equivalence between MLE and Entropic OT; and (2) revealing EM’s intrinsic structure as a geometrically principled optimization path over an OT loss—thereby offering novel geometric and variational interpretations for mixture modeling. This bridges statistical inference and optimal transport theory, enabling cross-methodological insights.

Technology Category

• Machine Learning: Information Theory
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This note aims to demonstrate that performing maximum-likelihood estimation for a mixture model is equivalent to minimizing over the parameters an optimal transport problem with entropic regularization. The objective is pedagogical: we seek to present this already known result in a concise and hopefully simple manner. We give an illustration with Gaussian mixture models by showing that the standard EM algorithm is a specific block-coordinate descent on an optimal transport loss.

Problem

• Mixture Models
• Maximum Likelihood Estimation
• Optimal Transport

Innovation

• Entropy-Regularized Optimal Transport
• Gaussian Mixture Model
• Maximum Likelihood Estimation