Momentum SVGD-EM for Accelerated Maximum Marginal Likelihood Estimation

2026-03-09

AI Summary

This work addresses the slow convergence of maximum marginal likelihood estimation (MMLE) in high-dimensional and complex tasks by reinterpreting the EM algorithm as a coordinate descent procedure in the joint space of model parameters and probability measures. It introduces, for the first time within the SVGD-EM framework, a Nesterov momentum mechanism to simultaneously accelerate both parameter updates and probability measure optimization. By integrating Stein variational gradient descent, free-energy functional optimization, and interacting particle dynamics, the proposed method substantially enhances convergence efficiency. Experimental results demonstrate that across a range of tasks—from low- to high-dimensional settings—the approach significantly reduces the number of required iterations and achieves markedly faster convergence and improved practicality compared to existing methods.

Technology Category

• Machine Learning: Optimization
• Search and Optimization: Mixed Discrete/Continuous Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Maximum marginal likelihood estimation (MMLE) can be formulated as the optimization of a free energy functional. From this viewpoint, the Expectation-Maximisation (EM) algorithm admits a natural interpretation as a coordinate descent method over the joint space of model parameters and probability measures. Recently, a significant body of work has adopted this perspective, leading to interacting particle algorithms for MMLE. In this paper, we propose an accelerated version of one such procedure, based on Stein variational gradient descent (SVGD), by introducing Nesterov acceleration in both the parameter updates and in the space of probability measures. The resulting method, termed Momentum SVGD-EM, consistently accelerates convergence in terms of required iterations across various tasks of increasing difficulty, demonstrating effectiveness in both low- and high-dimensional settings.

Problem

• Maximum Marginal Likelihood Estimation
• Stein Variational Gradient Descent
• Expectation-Maximisation
• Acceleration
• Convergence

Innovation

• Momentum SVGD-EM
• Nesterov acceleration
• Stein variational gradient descent
• maximum marginal likelihood estimation
• interacting particle algorithms