A mirror descent approach to maximum likelihood estimation in latent variable models

2025-01-27

AI Summary

Standard Expectation-Maximization (EM) algorithms are ill-suited for statistical models with discrete latent variables due to their reliance on differentiability and continuous latent spaces. Method: We propose the first unified framework integrating Mirror Descent (MD) with Sequential Monte Carlo (SMC) for joint parameter estimation and posterior inference. Our approach jointly minimizes a variational functional over both the parameter space and the space of probability measures, enabling maximum likelihood estimation (MLE) without requiring latent variable continuity. Contribution/Results: This work breaks EM’s dependence on latent variable continuity, marking the first application of MD to MLE in discrete latent variable models, with rigorous convergence guarantees established. Experiments demonstrate significant improvements over standard EM across multiple discrete latent variable tasks; on real-valued latent variable benchmarks, our method matches state-of-the-art performance, validating both theoretical soundness and empirical robustness.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Search and Optimization: Mixed Discrete/Continuous Search
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

We introduce an approach based on mirror descent and sequential Monte Carlo (SMC) to perform joint parameter inference and posterior estimation in latent variable models. This approach is based on minimisation of a functional over the parameter space and the space of probability distributions and, contrary to other popular approaches, can be implemented when the latent variable takes values in discrete spaces. We provide a detailed theoretical analysis of both the mirror descent algorithm and its approximation via SMC. We experimentally show that the proposed algorithm outperforms standard expectation maximisation algorithms and is competitive with other popular methods for real-valued latent variables.

Problem

• Performs joint parameter inference and posterior estimation in latent variable models
• Enables estimation when latent variables take discrete values
• Outperforms standard expectation maximisation algorithms in performance

Innovation

• Mirror descent for maximum likelihood estimation
• Sequential Monte Carlo for posterior estimation
• Handles latent variables in discrete spaces